Every programmer needs a way to write their code, whether using a general-purpose text editor like the one that comes bundled with your Operating System, a source code editor like VS Code or an Integrated Development Environment (IDE) like one of those from JetBrains. The one thing in common with all these editors I’ve mentioned is that they have a graphical user interface. However, there’s another family of editors that are terminal-based, for example, Vim, GNU Emacs and others like GNU nano.

I think my first exposure to these terminal-based editors was when I needed to edit files while working from the terminal, probably in my early Linux days, and most likely when setting up Linux VPSes and editing config files. Most Ubuntu VPS setup tutorials typically mention nano, as I understand it’s available by default without requiring additional installation.

I can’t remember how I found myself using Vim, but I do know that I preferred it to GNU nano! To this day, I always install Vim on any machine I’m using — not because I use it as my primary editor, but because that’s what I wanna use when I need to edit files while working in the terminal.

I know the Vim basics, having gone through vimtutor a while ago. I have always wanted to know Vim better, and be able to comfortably and productively use it for day-to-day work. It requires putting in a bit of work, and constant practice in order to build muscle memory. I just never quite got to doing this.

So many editors

My first programming language was Java, and I used Netbeans IDE for it, though I also experimented with Eclipse. When I got into building WordPress websites and tinkering with PHP, I used Notepad++, being on a Windows machine at the time. However, I later switched to Atom, and also experimented with Brackets. By the time I was getting into python programming, I liked Sublime Text, so I used it for a while. Then I discovered VS Code, and decided to switch to it! I was used to the Sublime Text key bindings, so the Sublime Text Keymap and Settings Importer extension was always handy. I still kept Sublime Text because I felt that it was more lightweight and faster, so I’d use it when I needed to make one-off edits, or just as a general purpose editor.

So VS Code has remained my main code editor, even though I'd have more lightweight editors lying around — I have since replaced Sublime Text with Zed, which I use when I'm not working on a specific project, e.g when I want to scribble something down or edit a file quickly.

Talk is cheap …

Alright, back to the Vim discussion! As I mentioned earlier, it's always been at the back of my mind to dig deeper into it and get more comfortable with it, as part of my own personal development as a programmer.

The thing is, I enjoy working from the terminal, but there are lots of important things I don't know well that would make me more productive in this environment, where I interact with files in one way or the other. Being proficient with Vim is one of those things.

Now, couple of days before writing this post, I started looking into resources I could use to up my Vim game. I bookmarked the following, with a plan to go through them and practice daily:

• https://github.com/iggredible/Learn-Vim

• https://swedishembedded.com/developers/vim-in-minutes

• https://coffeeaddict.dev/why-nvim/

Alongside this, I also bookmarked some tmux resources, because I think it’s an equally important tool

• https://thoughtbot.com/upcase/tmux

• https://www.redhat.com/en/blog/introduction-tmux-linux

• https://hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/

However, before I could dive deep into these resources, another programmer, in a local WhatsApp group for programmers, suggested trying out Neovim, specifically the AstroNvim distribution. At this point, I'd already come across Neovim, and seen lots of folks on YouTube using it, but never really got to check it out. This time, I decided to look into Neovim.

I checked out the AstroNvim website, and I was amazed! Before diving in though, I decided to find out more on YouTube, so I came across this video, which compared various Neovim distributions:

Now I was presented with even more options, all of which looked awesome! Throughout the video, one thing that I particularly liked was which-key, a plugin that shows you available keybindings when you pause mid-command. This really blew my mind! Anyway, the YouTube algorithm then brought up the following video, where ThePrimeagen goes through an article “Beginners should use a preconfigured Neovim distribution“, and gives his take.

ThePrimeagen strongly advocates for Kickstart as the ideal starting point for new Neovim users, preferring it over heavier, pre-configured distributions like LunarVim or AstroNvim.

His advice can be broken down into the following key points:

• The Best "First Step": ThePrimeagen believes Kickstart is a "really great first step" because it provides exactly what a user needs without being overwhelming. He contrasts this with full distributions, where "diving in deep" can be bewildering due to the sheer amount of configuration and abstraction involved.

• Minimal and Educational: He praises Kickstart for being contained within a single file and acting as a "constrained version" of a setup. This simplicity makes it a "Launchpad" for users to build their own unique configuration, rather than getting stuck in someone else's ecosystem.

• Avoids Complexity: ThePrimeagen argues that with heavy distributions, understanding how to make edits or fix issues requires a "huge leap" in knowledge regarding project structure, Lua, and plugin management (e.g., understanding init, after, plugin, etc.). Kickstart avoids this barrier to entry.

• Kickstart vs. Distros: ThePrimeagen holds the "opposite take" to the idea that beginners must use complex distros. He suggests that pre-configured distributions (like LazyVim) are better suited for users who already "know it all," are tired of configuring their own editor, and just want a working environment. For those actually starting their journey, he insists: "start off with Kickstart".

I resonated with ThePrimeagen’s arguments, because my current Vim setup is very minimal, and I have good control over it, compared to the time when I used the Janus Vim distribution, which seems to no longer be maintained. I was convinced to go with the Kickstart approach, as it felt like the right balance between "works immediately" and "I understand what's happening."

Here we go

I found that I didn’t actually have to uninstall vim in order to use Neovim, because Neovim has its own configuration, and you launch it via nvim, so it can coexist with vim.

On Fedora, installation was straightforward:

sudo dnf install -y neovim python3-neovim

Setting up Kickstart was simple:

# Clone kickstart
git clone https://github.com/nvim-lua/kickstart.nvim.git ~/.config/nvim

# Start neovim - plugins auto-install on first run
nvim

First Customizations

Theme: Catppuccin Mocha

I love Catppuccin, I use it wherever it’s supported (which is almost everywhere!), so I wanted to keep my Catppuccin Mocha theme from my Vim setup. In Kickstart, this meant replacing the default tokyonight theme. I found the colorscheme plugin section in init.lua and swapped it out:

{
  'catppuccin/nvim',
  name = 'catppuccin',
  priority = 1000,
  config = function()
    require('catppuccin').setup {
      flavour = 'mocha', -- latte, frappe, macchiato, mocha
      transparent_background = false,
      integrations = {
        blink_cmp = true,
        gitsigns = true,
        nvimtree = true,
        treesitter = true,
        telescope = true,
        mason = true,
        which_key = true,
      },
    }
    vim.cmd.colorscheme 'catppuccin'
  end,
},

Enabling Nerd Font Support

My Fedora setup script(s) already installed JetBrains Mono Nerd Font and Fantasque Sans Mono Nerd Font. To enable the fancy icons in Neovim, I just needed to flip one setting:

vim.g.have_nerd_font = true  -- Changed from false

Adding Neo-tree File Explorer

Kickstart is minimal by design—it doesn't even include a file explorer by default! It expects you to use Telescope for navigation (Space + s + f), which is actually faster once you get used to it.

But coming from VSCode, I wanted a traditional tree view available. Kickstart includes an optional Neo-tree plugin that's commented out. I just uncommented this line:

require 'kickstart.plugins.neo-tree',

Now I can toggle the file tree with \ (backslash).

Fixing Language Server Names

I hit my first real error when trying to configure language servers. The Mason package names don't always match what you'd expect. I tried to install lua_ls but got an error:

Cannot find package "lua_ls"

The correct name is lua-language-server (hyphens, not underscores). This pattern held for other servers too:

ensure_installed = {
    'lua-language-server',  -- Not lua_ls
    'stylua',
    'pyright',              -- Python
    'gopls',                -- Go
    'bash-language-server', -- Shell scripts
}

Pro tip: Run :Mason in Neovim to browse all available packages and see their exact names.

Understanding Which-Key

As mentioned earlier, one of the best discoveries I made was which-key.

Press Space and wait ~300ms? A popup shows all leader key commands. Press g and wait? See all the g commands like gd (go to definition).

It makes Neovim self-documenting. You don't need to memorize everything upfront—just start typing and which-key shows you what's possible. Sweet!

The Health Check

Neovim includes a built-in health check system: :checkhealth

This was useful for validating my setup. The output showed:

Everything important working:

• LSP configured correctly

• Treesitter parsers installed (Python, Go, Dockerfile, JSON, YAML, etc.)

• Telescope fuzzy finder ready

• All language servers installed

Warnings I could safely ignore:

• Missing support for languages I don't use (PHP, Java, Julia)

• Optional providers (Node.js, Perl, Ruby) - only needed for specific plugins

• tree-sitter-cli not installed - even though it’s really only required if you're developing parsers, I decided to install it, but encountered an error due to missing C headers

Installing tree-sitter (I already have Rust set up on my machine):

cargo install tree-sitter-cli

But it failed due to compilation errors which I resolved by installing the missing development packages:

sudo dnf install clang-devel libstdc++-devel

After that, cargo install tree-sitter-cli completed successfully, though, in hindsight, I could have skipped this entirely since it's only needed for parser development, not everyday use.

Kickstart's health check warned about the missing neovim npm package. Instead of using npm install -g, I used Volta (which I already use for Node.js version management):

volta install neovim

This keeps everything consistent with my existing toolchain.

Reflections: Why Bother?

This isn't about proving anything or cargo-culting what "real developers" use. It's about finding a workflow that matches how I think and work.

For me, that means having powerful text manipulation at my fingertips, and working in a fast, lightweight environment. I also want to understand my tools deeply instead of treating them as black boxes.

I'm glad I got started with Neovim via Kickstart, I think it was the right call. The single-file approach meant I could read and understand my entire config. Not abandoning my old vim setup removed the pressure, I had a fallback if I got stuck.

:checkhealth immediately showed what was working and what needed attention. Turns out most things just worked. LSP, completion, fuzzy finding, git integration, all there from the start. And it was noticeably snappier than VSCode, even with plugins.

The learning curve wasn't as steep as I thought. With which-key and a good starter config, discovery is built into the workflow. You don't have to memorize keybindings, you learn them by seeing them.

Neovim with Kickstart gets me there while keeping the learning curve manageable. The setup took maybe an hour, but I already have a fully functional development environment that rivals VSCode's capabilities for my stack.

What's Next

Now I actually have to use it. Daily. On real projects. I want to learn the navigation patterns deeply, especially text objects. I'll add plugins gradually as I discover needs, not pre-emptively.

I need to get comfortable with Telescope for navigation instead of always reaching for the file tree. And I want to explore the debugging support (nvim-dap) once I hit something I can't solve with print statements.

The Problem with Traditional CV Management

Like many professionals, I started my career using Microsoft Word for my CV. As a civil engineer, this seemed perfectly adequate at first. However, the problems quickly became apparent: multiple versions scattered across my computer, formatting inconsistencies, and the constant struggle to maintain a coherent document history. Which file had that perfect project description I wrote a couple of months ago?

The problem wasn't just about managing versions. As someone who values efficiency and automation, the traditional word processor approach felt increasingly at odds with modern development practices. I wanted something that would:

• Allow version control like my code

• Support easy formatting without fighting with Word's layout engine

• Generate a high quality PDF

• Enable quick updates without needing a Word processor or dealing with complex formatting

The LaTeX Era

In August 2019, I made my first attempt at modernizing my CV workflow by moving to LaTeX. Using the moderncv package, I created a professionally formatted document and, more importantly, put it under version control in a private GitLab repository. This was a significant improvement – my CV was now in plain text, versioned, and generated a consistent PDF output with professional typesetting.

However, LaTeX came with its own set of challenges:

• The TeX environment setup was non-trivial

• Sometimes, making customizations meant diving deep into LaTeX internals

• The development cycle was slow: edit, compile, check PDF, repeat

Career Transition and New Requirements

When I began contemplating a switch to tech in 2021/2022, I realized I needed to update my CV and tailor it to suit a tech role. I still wanted to maintain my Civil Engineering focused CV. While doing some research on writing tech-centric CVs, I realised I needed to do more than just updating my CV -- I needed to completely rewrite it and structure it in a different way. I didn't want to use LaTeX, so I had to figure out another way. My research led me to explore various alternatives, including JSON Resume and Prateek Agarwal's json_resume. The idea was appealing: structured data in JSON format, separate from presentation concerns.

While JSON Resume is an excellent project, I couldn't find templates that matched my vision, and I lacked the time and skills to create my own. I settled on json_resume, but found myself still not entirely satisfied. My solution evolved into a web project with hand-customized HTML templates. The workflow was functional but hacky:.

• lite-server would serve the resume webpage

• Puppeteer would generate PDFs from the served page

• Changes required editing HTML and manual PDF regeneration

While this solved some problems, it introduced new ones. The setup was complex, not user-friendly, and felt more like a temporary hack than a sustainable solution. Surely, there has to be a better way.

The Markdown Renaissance

By 2024, frustration with my existing setup led me back to the drawing board. I discovered Eliseo Papa's markdown-cv project, which resonated with my goals of simplicity and flexibility: simple content authoring with the ability to produce both PDF and web outputs. However, I wanted more control over styling and didn't want to depend on Ruby/Jekyll.

Initially, I got caught in the trap of overthinking the solution. I considered using 11ty, a static site generator I was already familiar with and very fond of, but found myself in development hell – planning features without writing code.

The breakthrough came in September 2024 when I decided to strip away the complexity and focus on solving the core problem. I chose Vite for its simplicity and modern development experience, avoiding frameworks that might constrain the solution.

This led to my current project, which achieves several key goals:

1. Simple Content Authoring: A single markdown file (with YAML frontmatter) for all CV content -- parsed using Marked

2. Modern Development Experience: Built with Vite for fast development and building

3. Automated PDF Generation: CI/CD pipeline that automatically generates PDFs using Puppeteer and optionally deploy a static site if I needed to link to an html version of my CV.

4. Framework Independence: Vanilla JavaScript and SCSS, no frameworks

5. Modern Styling: Using Phosphor Icons and responsive design

The architecture is intentionally simple:

1. Write CV content in Markdown with YAML frontmatter

2. Parse and transform to a responsive webpage

3. Generate PDF through Puppeteer

4. Automate builds and deployments through CI

Technical Highlights

The project taught me several valuable lessons:

1. Sometimes Less is More

Initially, I overthought the solution, considering frameworks like 11ty. By stepping back and focusing on core functionality, I created something more maintainable and useful.

2. Expanding Technical Skills

As a Python developer, this project pushed me to improve my JavaScript and frontend development skills. I worked with:

• Vite for building and development

• Vitest for testing

• PDF-lib for PDF manipulation

• SCSS for styling, including defining print styles

3. CI/CD Integration

The automated PDF generation through GitHub Actions and GitLab CI/CD made the tool truly practical for regular use.

Status Quo and Looking Forward

The current solution achieves my primary goals:

• Single source of truth in Markdown

• Version-controlled content

• Automated PDF generation

• Easy customization

• Responsive web output

• Simple development workflow

While the tool isn't perfect, it solves the real-world problem of maintaining a professional CV in a developer-friendly way. The journey from Word to this current solution reflects not just my personal growth as a developer but also the evolution of web technologies and development practices.

If you find this useful or have suggestions for improvement, I'd love to hear from you! The project is now open-source and available for anyone facing similar challenges in managing their CV. I'm sharing it with the community in hopes that others might find it useful or even contribute improvements.

Remember, sometimes the best solution isn't the most sophisticated one - it's the one that solves your problem while being maintainable and enjoyable to use.

Wanna try it out? Check out the GitHub repository for a demo, installation instructions and documentation.

I was constantly getting annoyed by this. Every time this happened, I'd end up with terminal application windows scattered across my desktop, or I'd resort to manually opening a new tab and typing cd /path/to/long/directory/name. Neither solution was efficient. I decided to do something about it.

Kowalski, options!

I searched online and found several approaches:

• Nautilus scripts are a powerful feature. This post on Fedora Magazine gives a practical example of converting images to a particular format. I also found another good example by Neil Brown here. However, for my particular use case, I didn't want to have to right-click, then go to "Scripts", then "Open in Console Tab". I wanted a top-level entry, not something hidden in a sub-menu.

• Nautilus actions, which was deprecated and renamed to FileManager Actions, then later archived because there were no contributions for several years. If you are interested in the details, please see this askubuntu.com post, this one and this one. Martin Bartlett has written a full-function replacement for this extension that works with the current version of Gnome Files. It's called Actions For Nautilus and it's at https://github.com/bassmanitram/actions-for-nautilus. As my use case was a very simple one, I thought this route was going to be too complex for my needs. So I continued digging through the rabbit hole that is the internet!

• Nautilus Python, an extension for Nautilus that allows further extending it with Python scripts with the help of Nautilus’s GObject API. I think I may have stumbled upon it via a Reddit post Is there a replacement for Nautilus-actions / Filemanager-actions?. This is an official GNOME project, you can check out the project's git repository.

As an aside, when I asked Claude for a solution to my problem, the initial response I got was to use Nautilus actions. After raising the issue of it being deprecated, Claude then suggested using Nautilus scripts, which I didn't want. It was only then that Claude suggested Nautilus python (among others which included FileManager Actions and creating a custom GNOME extension).

And, we have a winner!

I decided to go with Nautilus Python, as it seemed to be the most practical solution for my use case, and was a perfect fit for me as a Python developer.

Procedure

1. First, install the Nautilus Python extension. This is for Fedora. If you're using a different distro, please adapt accordingly.

sudo dnf install nautilus-python

2. Create the extensions directory (if it doesn't exist):

mkdir -p ~/.local/share/nautilus-python/extensions/

3. Create a new file in the extensions directory. In my case, I called it ptyxis_tab_extension.py, because Ptyxis is the default terminal application in Fedora (as of Fedora 41), replacing the long-standing GNOME Terminal. This change came after a Fedora Workstation discussion about adopting this newer terminal emulator, as shared in this Reddit thread.

touch ~/.local/share/nautilus-python/extensions/ptyxis_tab_extension.py

4. Write your code. You can see some examples here. Here's the code for opening a new ptyxis terminal tab at the current location:

#!/usr/bin/env python3

import subprocess
from gi.repository import Nautilus, GObject


class PtyxisTabMenuProvider(GObject.GObject, Nautilus.MenuProvider):
    def __init__(self):
        pass

    def get_file_items(self, files):
        # Only show for folders
        if len(files) != 1 or not files[0].is_directory():
            return []

        item = Nautilus.MenuItem(
            name="PtyxisTabExtension::open_in_tab",
            label="Open in Console Tab",
            tip="Open the folder in a Ptyxis console tab",
        )

        item.connect("activate", self.open_in_ptyxis_tab, files[0])
        return [item]

    # Handle right-click on background (empty space)
    def get_background_items(self, current_folder):
        item = Nautilus.MenuItem(
            name="PtyxisTabExtension::open_current_in_tab",
            label="Open in Console Tab",
            tip="Open this folder in a Ptyxis console tab",
        )

        item.connect("activate", self.open_current_in_ptyxis_tab, current_folder)
        return [item]

    def open_in_ptyxis_tab(self, menu, file):
        filepath = file.get_location().get_path()
        subprocess.Popen(["ptyxis", "--tab", "-d", filepath])

    def open_current_in_ptyxis_tab(self, menu, folder):
        filepath = folder.get_location().get_path()
        subprocess.Popen(["ptyxis", "--tab", "-d", filepath])

5. Make the file executable:

chmod +x ~/.local/share/nautilus-python/extensions/ptyxis_tab_extension.py

6. Restart Nautilus:

nautilus -q

7. Enjoy!

Verba Finalia

With very few lines of code, we have an extension that adds the "Open in Console Tab" option in two places:

• When right-clicking on a folder

• When right-clicking in the empty space within a directory

When clicked, it runs ptyxis --tab -d /path/to/folder which opens the selected directory in a new tab in your existing Ptyxis window (or open a New window if there isn't any).

How cool is that? No more scattered terminal windows or tedious directory typing! Now I can quickly navigate my file system and open directories in terminal tabs with just a right-click.

The current implementation works well, but there are a few potential enhancements:

• Positioning the menu option closer to the original "Open in Console" option

• Adding an icon to match the Ptyxis style

But that's something for another day! For now, my problem is solved and I am happy!

When I upgraded from Fedora 40 to 41, I didn't expect it to lead me down a path of discovery in Linux package development. The upgrade brought several changes, but the most impactful was the complete shift to Wayland from X11. As someone who regularly used screen recording tools, this change immediately affected my workflow - I suddenly couldn't use my trusty Peek screen recorder, which relied on X11.

The Search for Alternatives

After some searching, I found Kooha, a modern screen recorder designed with Wayland in mind.

Perfect! But there was a catch: it was only available as a Flatpak. For those unfamiliar, Flatpak is a package format that bundles an application with all its dependencies, making it work across different Linux distributions. While this is excellent for compatibility, it came with significant overhead:

Flatpak Installation:
- GNOME Platform (46): 364.0 MB
- GNOME Platform Locale: 379.2 MB
- GL/Graphics drivers: 350.2 MB
  • GL default: 168.3 MB
  • GL default extra: 168.3 MB
  • VAAPI Intel: 13.6 MB
- Codecs (openh264): 976.5 KB
- Application Locale: 125.4 KB
- Kooha application itself: 2.2 MB
Total: ~1.1 GB

Native RPM Package:
- Package size: 2.2 MB
- Uses existing system libraries
Total: 2.2 MB

For someone mindful of disk space, this felt like overkill for a simple screen recorder. My system already had most of these dependencies installed. Why download them again?

Taking the Plunge into Package Building

That's when I had an idea: why not build Kooha natively for Fedora? With zero RPM packaging experience but plenty of curiosity, I set out to create a native Fedora package. Thankfully, with the help of claude.ai, I was able to do this in under an hour.

The first challenge was creating a spec file - the recipe that tells the RPM build system how to create the package. Here's what I learned:

# Basic information about the package
Name:           kooha
Version:        2.3.0
Release:        1%{?dist}
Summary:        Elegantly record your screen

# Licensing and meta
License:        GPL-3.0
URL:            https://github.com/SeaDve/Kooha
Source0:        %{url}/archive/refs/tags/v%{version}.tar.gz

# Build requirements
BuildRequires:  meson
BuildRequires:  ninja-build
BuildRequires:  cargo
BuildRequires:  rust
BuildRequires:  appstream
BuildRequires:  pkgconfig(gstreamer-1.0)
BuildRequires:  pkgconfig(gstreamer-plugins-base-1.0)
BuildRequires:  pkgconfig(gtk4)
BuildRequires:  pkgconfig(libadwaita-1)
BuildRequires:  pkgconfig(glib-2.0)

# Runtime requirements
Requires:       pipewire
Requires:       gstreamer1-plugins-base
Requires:       gstreamer1-plugins-ugly-free
Requires:       gstreamer1-plugins-bad-free
Requires:       pipewire-gstreamer
Requires:       xdg-desktop-portal
Requires:       gtk4
Requires:       libadwaita

%description
Kooha is a simple screen recorder with a minimal interface. Record your screen
in an intuitive and straightforward way without distractions. Features include
recording microphone and desktop audio, support for WebM, MP4, GIF, and Matroska
formats, and the ability to select a monitor or portion of the screen to record.

%prep
%autosetup -n Kooha-%{version}

%build
%meson
%meson_build

%install
%meson_install

%check
appstreamcli validate %{buildroot}/%{_datadir}/metainfo/*.metainfo.xml

%files
%license COPYING
%doc README.md
%{_bindir}/%{name}
%{_datadir}/applications/*.desktop
%{_datadir}/metainfo/*.metainfo.xml
%{_datadir}/icons/hicolor/*/apps/*.svg
%{_datadir}/glib-2.0/schemas/*.gschema.xml
%{_datadir}/locale/*/LC_MESSAGES/*.mo
%{_datadir}/dbus-1/services/*.service
%{_datadir}/%{name}/resources.gresource

%changelog

This spec file tells the RPM build system everything it needs to know about the package: what it's called, what version we're building, what dependencies it needs, and how to build it. Getting the dependencies right was particularly tricky - I had to understand both what was needed to build the application (BuildRequires) and what was needed to run it (Requires). With the spec file in place, I could then set up the build environment:

# Setting up the RPM build environment
rpmdev-setuptree

# Installing build dependencies
dnf install rpmdevtools rpm-build meson ninja-build \
    gstreamer1-devel gtk4-devel libadwaita-devel

# Building the package
rpmbuild -ba kooha.spec

What followed was a fascinating journey into the heart of Linux packaging. Along the way, I discovered three things that changed how I think about Linux applications:

First, I gained a deep appreciation for the freeDesktop.org ecosystem. Building a native Linux desktop application isn't just about the code - it's about integrating with a rich framework of standards and specifications that make the Linux desktop experience seamless.

Second, package management turned out to be an intricate dance of dependencies. Understanding the relationship between build-time and runtime requirements, and how they all fit together, was both challenging and enlightening.

Finally, I came face-to-face with the challenges of cross-distribution compatibility - which gave me a whole new appreciation for why the original author chose Flatpak in the first place.

Automating the process

I decided to not end at just building the application locally. I wanted to automate the process so that

• I had a way to ensure that I had the latest updates

• I had a reference in case I (or others) encountered a similar challenge in future

• Others could benefit from it, and perhaps adapt it to other distros

I therefore went ahead and built an automated RPM package builder for Kooha. The automation process involved creating GitHub Actions workflows to:

• Check for new Kooha releases periodically

• Build fresh RPM packages when updates are available

• Create GitHub releases with the built packages

The result was a fully automated system that keeps the RPM package updated with minimal maintenance overhead. The RPM packages are available on the project's GitHub releases page. If you're interested in contributing or learning more about RPM packaging, the project's README provides detailed information to get you started.

Learning Through Building

This project taught me so much about Linux application development:

• The complexities of building for different distributions

• The role of desktop integration through FreeDesktop.org standards

• The importance of automated builds and release processes

• The beauty of being able to customize and build software to suit your needs

The Sustainability Challenge

Working on this project opened my eyes to the challenges of maintaining open source software. What looks simple on the surface often requires significant ongoing effort:

• Continuous Updates: Every new OS release, dependency update, or security patch requires testing and potentially updates

• User Support: Responding to issues, helping with installation problems, and documenting solutions

• Infrastructure Costs: Build systems, hosting, and distribution all have associated costs

• Time Investment: Maintainers often work on these projects in their spare time

This brings us to an important point: open source software needs financial support to thrive. While the software is free to use, its development and maintenance aren't free. There are several ways to support the open source ecosystem:

1. Direct Support:

   • Donate to projects you use regularly

   • Subscribe to maintainers' GitHub Sponsors or Patreon accounts

   • Pay for support or additional features when offered

2. Indirect Support:

   • Report bugs and provide detailed feedback

   • Contribute documentation or translations

   • Share and promote projects you find valuable

   • Help other users in community forums

The Power of Open Source

What strikes me most about this journey is how it exemplifies the power of open source. When faced with a problem, I had the freedom to:

1. Examine how the original application was built

2. Create a different distribution method

3. Share my solution with others

4. Learn about Linux packaging in the process

While Flatpak is an excellent solution (and I now understand why the Kooha developer chose it), having alternatives is what makes open source special. Someone might prefer the Flatpak version for its portability, while others might want an RPM (or deb, etc.) version for its smaller footprint.

Most importantly, this experience taught me that sometimes the best way to learn is to dive in and build something, even if you've never done it before. The Linux ecosystem might seem complex, but it's also incredibly rewarding to explore and contribute to.

Getting Involved

If you're inspired to explore Linux package development:

1. Start Small: Pick a simple application you use and try building it from source

2. Learn the Tools: Familiarize yourself with packaging tools like rpmbuild or debuild

3. Join Communities: Participate in distribution-specific packaging groups

4. Support Projects: Consider supporting Kooha and other open source projects you use

Have you built packages for Linux before? What was your experience like? Please share your thoughts in the comments below!

This is where atomic updates come in. By updating one package at a time and committing each change separately, you get fine-grained control over your dependencies. In this post, I’ll demonstrate how to automate this process with a Bash script.

Why Automate the Process?

Manually updating packages with poetry update <package-name> can be time-consuming and repetitive, especially if your project has several outdated dependencies. Imagine running this command for each of 25 packages, and then manually committing each change. It becomes tedious, error-prone, and slows down your workflow.

Here’s why automating the process is important:

• Efficiency: Instead of running individual update commands and making commits for each, the script automates the entire process in a loop. It checks for outdated packages, runs the update, commits the changes, and moves on to the next—without manual intervention.

• Consistency: Automation ensures that all updates follow a consistent process. You won’t forget to commit a change or make inconsistent commit messages—everything is handled systematically by the script.

• Focus on What Matters: Rather than spending your time running commands, you can focus on reviewing important changes, like release notes or resolving breaking changes introduced by updates. Let the script handle the grunt work.

The Developer's Responsibility

While automation makes dependency updates easier, it’s still the developer’s responsibility to ensure compatibility. Your project’s pyproject.toml file should specify dependency version constraints that prevent major breaking changes. For example, specifying something like:

wagtail = ">=6.1,<6.2"

ensures that when you update Wagtail, only compatible versions within this range will be installed. This minimizes the risk of major updates that could break your project.

Even with these safeguards, it’s essential to thoroughly test your project after updating dependencies to ensure nothing breaks. Automation can save you time, but you need to verify that everything works as expected.

Tools Like Renovate and Dependabot

There are dedicated tools like Renovate and Dependabot that help automate dependency updates by creating pull requests whenever an update is available. These tools integrate with GitHub and other platforms, providing a more structured way to handle updates.

However, there are times when such tools might not be available or suitable for the project you're working on—whether due to limitations in the development environment or specific project needs. This is where having a custom solution like a custom Bash script can be particularly useful.

The Script

Here’s the Bash script I wrote

#!/bin/bash

echo "Checking for outdated packages..."

# Checking outdated packages with Poetry
outdated_packages=$(poetry show --outdated | awk '{print $1, $2, $3}')

echo "Starting the update process..."

# Read the outdated packages list line by line
echo "$outdated_packages" | while read -r line; do
    read -ra ADDR <<<"$line"
    package_name=${ADDR[0]}
    current_version=${ADDR[1]}
    latest_version=${ADDR[2]}

    # Update package
    echo "Attempting to update $package_name from $current_version to $latest_version..."
    if poetry update "$package_name"; then
        git add poetry.lock
        git commit -m "🐍📦 Update $package_name ($current_version -> $latest_version)"
        echo "Successfully updated and committed $package_name"
    else
        echo "Failed to update $package_name. Continuing to next package..."
    fi

done

echo "Dependency update process completed."

How It Works

1. Check for outdated packages: The script starts by listing all outdated packages using poetry show --outdated. This command outputs the package name, current version, and the latest available version.

2. Update each package: For each outdated package, the script runs poetry update <package-name>. If the update succeeds, the changes are committed to Git with a message that clearly shows the version upgrade (e.g., 🐍📦 Update requests (2.25.0 -> 2.32.3)).

3. Handle errors: If an update fails (e.g., due to incompatibility issues), the script skips that package and moves on to the next, allowing the rest of the updates to proceed.

Why Atomic Updates Matter

The key advantage of this approach is that it provides granularity. When you commit each update individually, you can easily revert just one problematic package update without affecting others. For instance, if requests v2.32.3 breaks your project, you can roll back only that update, while keeping other updates intact.

Here’s how you would revert a single update:

git revert <commit-hash>

This undo command creates a new commit that reverses the changes from the specified commit—leaving the rest of your updates untouched.

Conclusion

By using this Bash script, you can keep your Poetry dependencies up to date while minimizing the risk of breaking changes. The use of atomic commits ensures that each update is easily traceable, and if something goes wrong, you have the power to roll back only the problematic update.

If you have access to tools like Renovate or Dependabot, they can provide even more automation and control over your updates. However, for projects where those tools aren’t available, this script provides a powerful and flexible alternative.

I've created a GitHub Gist with the full script—feel free to try it out in your own project!

Firstly, some disappointing news – I never actually made significant progress with my learning on Coursera. Yes, I enrolled in a couple of programmes, and actually started going through the lessons and so on, but I struggled to do it consistently, and so I was always playing catch-up. In the end, I never got to complete any of those courses I had enrolled in, which is a shame. What excuse do I really have? I could say that the pressures and demands of life and work got the best of me, but don't we all have pressure from time to time? Even then, it's not like one is constantly under pressure – there are highs and lows in life, different seasons, just as the Preacher puts it in Ecclesiastes 3

For everything there is a season, and a time for every matter under heaven ...

So that would be a pretty lame excuse. As I write this and reflect, I think the truth is that I didn't set my priorities right, and wasn't focused enough to ensure that I dedicated the required amount of time and effort into this. The fact that I had just started a new career in tech should have been a good motivation for me to pursue this diligently, but alas, here I am, lamenting on my failures!

I should have done better, but I didn't. I think the lesson for me is to make every effort to ensure that I stick to all the commitments I have made. Sometimes we rush into making a commitment without carefully considering all the options on the table, and evaluating the feasibility of actually sticking to that commitment. However, in my case, I think I did spend some time carefully planning and thinking about studying Computer Science, so that's not the issue here. It's easy to make plans, but committing to those plans is another thing, and that's what I failed to do.

Now, as I delved deeper into the tech industry, I started re-evaluating my career goals and thinking about the future and where I wanted to be. You learn new things and unlearn some of the old stuff, the preconceived notions and ideas you had previously, and the misconceptions. When I realised that my self-directed approach wasn't working, I started to think about the next steps. What am I going to do? I was still keen on gaining a solid foundation in CS, not just learning a programming language or framework. So I went back to the drawing board and decided to reconsider enrolling into a University to study, despite my earlier thoughts and conclusions. This brings me to my second item – I decided that enrolling in a formal MSc Computer Science program was the best way forward. I have therefore enrolled into a MSc Computer Science programme at the University of Sunderland, UK.

As I said in my first post in this series, the biggest challenges with enrolling to a University are finances and time. This is why I considered the University of Sunderland programme, because of its balance between affordability and flexibility. On the financial side of things, it is generally much more affordable than its competitors, with flexible payment options. On the time aspect, I get to focus on just one module at a time (rather than say 2 or 3, as is typically the case with similar programmes), for about 7-8 weeks. Obviously, I have to manage my time wisely and ensure that I dedicate sufficient time to the programme so that I don't lag behind.

The programme is ...

designed for ambitious individuals who aren’t from a computer science background and who want to launch a new career in computer science or want to incorporate computer science expertise and knowledge into their current field as a means of accelerating their employability and career progression.

Further, it is ...

also open to professionals who work in computer science roles and want to gain an academic qualification to enhance their credentials and career prospects.

Which means that I am a suitable candidate, right?

Anyway, today is the first day of the programme, and I saw it fit to continue with this series on Computer Science, as I've been too quiet! As I embark on this new chapter, I’m both nervous and excited. Have you ever faced similar challenges in your learning journey? I'd love to hear your experiences or any advice you might have!

You probably already use Markdown to write your docs, and even if you don't often write docs (which you should be doing!), your project's README is most likely written in Markdown. I have become so used to writing in Markdown that I even use the markdown-here browser extension so I can write emails in Markdown right within the Gmail interface! As John Gruber said in his introduction to Markdown:

Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).

This is why I find it appealing — you just write, with less worries about layouts and formatting. Wouldn't it be cool if we could write our Django email templates in Markdown, and render nice looking emails without breaking a sweat? Well, we can! So let's dive in and see how we can get this done.

Suppose, in our Django project, we want to send notification emails whenever somebody registers for an event. We have a plain text email template as follows:

Howdy {{ user }},

This is to confirm that you have successfully registered for “{{ event.title }}”. Here are the details, for your reference:

-----------------------------
Event: {{ event.title }}
Description: {{ event.description }}
Date and Time: {{ event.date_and_time }}
Location: {{ event.location }}
-----------------------------

We look forward to seeing you there!

Cheers,

— The Project Team

This is a Django template, with context variables user and event. This template is rendered using the render_to_string function in the django.template.loader module. Here's a custom send_mail function, where this is done:

from typing import List, Optional

from django.conf import settings
from django.core.mail import EmailMessage
from django.template.loader import render_to_string


def send_email(
    subject: str,
    to_email_list: List[str],
    template: str,
    context: Optional[dict] = None,
) -> None:
    """
    Sends an email with the specified subject,
    to the specified list of email recipients,
    using the supplied text template with optional context.

    Args:
        subject: The subject line of the email.
        to_email_list: A list of email addresses to send the email to.
        template: The path to the text template to use.
        context: A dictionary containing values to pass to the template (optional).

    Returns:
        None
    """

    # If context is None, set it to an empty dictionary
    if context is None:
        context = {}

    # Render the text template using the provided context
    text_content = render_to_string(template, context)

    # Create the email message object
    email = EmailMessage(
        subject=subject,
        body=text_content,
        from_email=settings.DEFAULT_FROM_EMAIL,
        to=to_email_list,
    )

    # Send the email
    email.send()

So, in order to send an email, we'd call our custom send_mail function like this:

send_email(
    subject=f"Event Registration for {event.title}",
    to_email_list=[user.email],
    template="events/event_registration_notification_email.txt",
    context={"event": event, "user": user},
)

And the user will get a plain text email like this:

Now, in order to bring Markdown into the mix, we need to make a couple of changes:

1. Write our template in Markdown

2. Install a Markdown parsing and rendering library

3. Modify our send_mail function to use the library above

So, let's get to it. Here's a slightly modified version of the template:

Howdy {{ user }},

This is to confirm that you have successfully registered for **{{ event.title }}**. Here are the details, for your reference:

---

- _Event_: {{ event.title }}
- _Description_: {{ event.description }}
- _Date and Time_: **{{ event.date_and_time }}**
- _Location_: {{ event.location }}

---

We look forward to seeing you there!

Cheers,

— The Project Team

Regarding a Markdown parsing and rendering library, the choice is entirely up to you! There are several options available, including Python-Markdown and Python-Markdown2, which are quite popular. These are Python implementations of John Gruber's original Perl-implemented Markdown. In my case, I wanted something that was based on GitHub's fork of cmark, so I could use GitHub Flavoured Markdown (GFM) out-of-the-box (without installing / configuring additional extensions), if I needed to. This led me to cmarkgfm and pycmarkgfm. I went with pycmarkgfm, which is similar to cmarkgfm, but with support for additional features such as task lists.

Install the library:

pip install pycmarkgfm

Now let's modify the send_mail function so we can make use of pycmarkgfm:

from typing import List, Optional

import pycmarkgfm
from django.conf import settings
from django.core.mail import EmailMessage
from django.template.loader import render_to_string


def send_email(
    subject: str,
    to_email_list: List[str],
    template: str,
    context: Optional[dict] = None,
    md_to_html: Optional[bool] = False,
) -> None:
    """
    Sends an email with the specified subject,
    to the specified list of email recipients,
    using the supplied text template with optional context.

    Args:
        subject: The subject line of the email.
        to_email_list: A list of email addresses to send the email to.
        template: The path to the text template to use.
        context: A dictionary containing values to pass to the template (optional).
        md_to_html: Whether the template content is written in Markdown format and
        should be rendered as HTML.

    Returns:
        None
    """

    # If context is None, set it to an empty dictionary
    if context is None:
        context = {}

    # Render the text template using the provided context
    text_content = render_to_string(template, context)

    if md_to_html:
        text_content = pycmarkgfm.gfm_to_html(text_content)

    # Create the email message object
    email = EmailMessage(
        subject=subject,
        body=text_content,
        from_email=settings.DEFAULT_FROM_EMAIL,
        to=to_email_list,
    )

    # If markdown to html is enabled, set the content type to HTML
    if md_to_html:
        email.content_subtype = "html"

    # Send the email
    email.send()

We have introduced a new Boolean md_to_html keyword argument, so that, when it's True:

• we apply pycmarkgfm's gfm_to_html function to the text rendered by django.template.loader.render_to_string, and

• change the email content_subtype to "html".

We could use the EmailMultiAlternatives class here, in order to send both text and HTML versions of our email. However, we'll keep things simple and just send the HTML version. You can check the Django docs for details of how to use the EmailMultiAlternatives class.

Here's what the email looks like

Pretty cool, right? We could end here and call it a day! However, we can make it even better, by adding a custom stylesheet. We can replicate the GitHub Markdown style by using Sindre Sorhus' github-markdown-css.

For a web page, we can easily use an external stylesheet via

<link href="example.css" rel="stylesheet" />

Or we could even embed the stylesheet via the <style> tag.

However, with emails, things are a little bit more complicated, mainly because

• most email clients do not support linking to external stylesheets due to security and privacy concerns

• email client CSS support for embedded stylesheets is limited (at the time of writing this post)

For more information about CSS in emails, please see the following resources, which I found quite helpful:

• https://myemma.com/blog/css-in-html-emails-what-you-need-to-know-to-get-started/

• https://mailtrap.io/blog/email-css/

• https://customer.io/blog/how-to-make-css-play-nice-in-html-emails-without-breaking-everything/

To ensure your emails render properly across different email clients, it is recommended to use inline CSS. That sounds like too much work! Fortunately, someone has written an excellent tool to do all this work for us. Enter premailer, by Peter Bengtsson!

From the project's README:

[Premailer] parses an HTML page, looks up style blocks and parses the CSS. It then uses the lxml.html parser to modify the DOM tree of the page accordingly.

Exactly what we need! Let's start by installing the package

pip install premailer

Next up, we'll need to download the github-markdown-css stylesheet and update our send_mail function to use premailer:

import os
from typing import List, Optional

import pycmarkgfm
from django.conf import settings
from django.core.mail import EmailMessage
from django.template.loader import render_to_string
from premailer import transform


def send_email(
    subject: str,
    to_email_list: List[str],
    template: str,
    context: Optional[dict] = None,
    md_to_html: Optional[bool] = False,
) -> None:
    """
    Sends an email with the specified subject,
    to the specified list of email recipients,
    using the supplied text template with optional context.

    Args:
        subject: The subject line of the email.
        to_email_list: A list of email addresses to send the email to.
        template: The path to the text template to use.
        context: A dictionary containing values to pass to the template (optional).
        md_to_html: Whether the template content is written in Markdown format and
        should be rendered as HTML.

    Returns:
        None
    """

    # If context is None, set it to an empty dictionary
    if context is None:
        context = {}

    # Render the text template using the provided context
    text_content = render_to_string(template, context)

    if md_to_html:
        html = pycmarkgfm.gfm_to_html(text_content)
        with open(
            os.path.join(settings.PROJECT_DIR, "assets/css/github-markdown.min.css"),
            "r",
            encoding="utf-8",
        ) as f:
            github_markdown_css = f.read()
        # see https://github.com/sindresorhus/github-markdown-css?tab=readme-ov-file#usage
        formatted_content = f"""
        <!DOCTYPE html>
        <html>
            <head>
                <meta name="viewport" content="width=device-width, initial-scale=1">
                <style>{github_markdown_css}</style>
                <style>
                    .markdown-body {{
                        box-sizing: border-box;
                        min-width: 200px;
                        max-width: 980px;
                        margin: 0 auto;
                        padding: 45px;
                    }}
                    @media (max-width: 767px) {{
                        .markdown-body {{
                            padding: 15px;
                        }}
                    }}
                </style>
            </head>
            <body class="markdown-body">{html}</body>
        </html>
        """
        text_content = transform(formatted_content)

    # Create the email message object
    email = EmailMessage(
        subject=subject,
        body=text_content,
        from_email=settings.DEFAULT_FROM_EMAIL,
        to=to_email_list,
    )

    # If markdown to html is enabled, set the content type to HTML
    if md_to_html:
        email.content_subtype = "html"

    # Send the email
    email.send()

Notes:

1. Asset Loading: We load the github-markdown-css stylesheet from a file (github-markdown.min.css), and embed the stylesheet via the <style> tag. The file can be located anywhere (it doesn't need to be within the STATICFILES_DIRS because it's not used on the frontend — we are only using it when generating emails). I minified the file using sass in order to reduce the file size.

2. HTML Content Generation: We create a formatted HTML string based on the example in the github-markdown-css README.

3. Inlining CSS Styles: The premailer.transform function is applied to the formatted HTML content to inline CSS styles.

This is the result:

Pretty sweet, right? Well, there you have it — decent-looking emails with minimal effort. After all, simple is better than complex!

Suppose we have a toys app and a ToyPage model with fields defined as shown below:

class ToyPage(Page):
    description = RichTextField(features=["bold", "italic", "link"])
    designer = models.ForeignKey(
        User, on_delete=models.SET_NULL, null=True, blank=True, related_name="toys"
    )
    use_designer_email = models.BooleanField(
        default=False,
        verbose_name="Use designer's email",
        help_text="Use the designer's email address as the contact email",
    )
    contact_email = models.EmailField(blank=True)

We want to show the contact_email field by default, and hide it when use_designer_email is True.

We can do this via JavaScript, by registering an event listener on the use_designer_email checkbox, and toggling the display of the use_designer_email FieldPanel.

Wagtail provides the insert_editor_js hook, which facilitates addition of extra JavaScript files or code snippets to the page editor. In our app's wagtail_hooks.py, we register a function with the above hook:

# /path/to/project/toys/wagtail_hooks.py
from django.templatetags.static import static
from django.utils.html import format_html
from wagtail import hooks


@hooks.register("insert_editor_js")
def editor_js():
    return format_html('<script src="{}"></script>', static("js/page-editor.js"))

We could write the JavaScript code directly in the editor_js function above, but for ease of maintenance I think it's better to just write the JavaScript code in its own file, which, in this case, is js/page-editor.js within the STATICFILES_DIRS. Now, before we write the JavaScript code, we need to use our web browser's developer tools to inspect the DOM so we understand the panel layout and get the correct selectors for the elements we'll be working with.

In the screenshot above, the element representing the use_designer_email BooleanField is highlighted. In this case, it's a checkbox input:

<input
  type="checkbox"
  name="use_designer_email"
  id="id_use_designer_email"
  aria-describedby="panel-child-content-child-designer-child-use_designer_email-helptext"
/>

And now, let's take a closer look at the contact_email:

The green rectangle represents the email input, while the red rectangle represents the containing div (with class w-panel__wrapper) which we actually want to hide/show — we don't want to hide just the input, but also the label and everything else associated with it. Here's the markup, for reference:

<div class="w-panel__wrapper">
  <label
    class="w-field__label"
    for="id_contact_email"
    id="id_contact_email-label"
  >
    Contact email
  </label>

  <div class="w-field__wrapper" data-field-wrapper="">
    <div
      class="w-field w-field--email_field w-field--email_input w-field--commentable"
      data-field=""
      data-contentpath="contact_email"
    >
      <div
        class="w-field__errors"
        data-field-errors=""
        id="panel-child-content-child-designer-child-contact_email-errors"
      ></div>

      <div
        class="w-field__help"
        id="panel-child-content-child-designer-child-contact_email-helptext"
        data-field-help=""
      ></div>

      <div class="w-field__input" data-field-input="">
        <input
          type="email"
          name="contact_email"
          maxlength="254"
          id="id_contact_email"
        />

        <button
          class="w-field__comment-button w-field__comment-button--add"
          type="button"
          data-component="add-comment-button"
          data-comment-add=""
          aria-label="Add comment"
          aria-describedby="id_contact_email-label"
        >
          <svg class="icon icon-comment-add icon" aria-hidden="true">
            <use href="#icon-comment-add"></use>
          </svg>
          <svg class="icon icon-comment-add-reversed icon" aria-hidden="true">
            <use href="#icon-comment-add-reversed"></use>
          </svg>
        </button>
      </div>
    </div>
  </div>
</div>

With this information, we can now go ahead and write our code:

document.addEventListener("DOMContentLoaded", function () {
  const checkbox = document.querySelector(
    'input[type="checkbox"][name="use_designer_email"][id="id_use_designer_email"]'
  );
  const emailField = document.querySelector(
    'input[type="email"][name="contact_email"][id="id_contact_email"]'
  );

  function toggleEmailField() {
    const panelWrapper = emailField.closest(".w-panel__wrapper");
    panelWrapper.style.display = checkbox.checked ? "none" : "block";
  }

  function initializePage() {
    checkbox.addEventListener("change", toggleEmailField);
  }

  initializePage();
});

A few things to note here:

1. Adding an event listener for the DOMContentLoaded event ensures that the code inside this function runs only when the DOM content is fully loaded and parsed.

2. The closest() method is applied on the emailField in order to find the closest ancestor element with the class "w-panel__wrapper" relative to the emailField. This is particularly important, because, as you may have noticed from the screenshots above, there are several divs with class "w-panel__wrapper", so we need to ensure that we are hiding/showing the correct one!

3. We dynamically toggle the visibility of the panelWrapper by adjusting it's style property to control the display CSS property. This involves switching between 'display: none;' and 'display: block;' based on the checked state of the checkbox.

4. We listen for the change event on the checkbox to trigger the toggleEmailField function when the checkbox state changes.

Here's what this looks like:

Pretty cool, right? Well, let's look at one more contrived example.

Now, going back to our ToyPage model, suppose we have an inline model on our ToyPage, and we want to control visibility of a field within the inline model. Here's the ToyCampaign inline model, and the updated ToyPage model:

class ToyCampaign(Orderable):
    page = ParentalKey("ToyPage", related_name="campaigns")

    title = models.CharField(max_length=255)

    start_date = models.DateTimeField()
    end_date_is_known = models.BooleanField(default=False)
    end_date = models.DateTimeField(blank=True, null=True)

    panels = [
        FieldPanel("title"),
        MultiFieldPanel(
            [
                FieldPanel("start_date"),
                FieldPanel("end_date_is_known"),
                FieldPanel("end_date"),
            ],
            heading="Dates",
        ),
    ]

    def clean(self):
        errors = defaultdict(list)
        super().clean()

        end_date = self.end_date

        if self.end_date_is_known and not end_date:
            errors["end_date"].append(
                _("Please specify the end date, since it is known!")
            )

        if end_date and end_date <= self.start_date:
            errors["end_date"].append(_("End date must be after start date"))

        if errors:
            raise ValidationError(errors)

    def __str__(self):
        return "Toy Campaign ‘{}’ on Page “{}”".format(self.title, self.page.title)


class ToyPage(Page):
    description = RichTextField(features=["bold", "italic", "link"])
    designer = models.ForeignKey(
        User, on_delete=models.SET_NULL, null=True, blank=True, related_name="toys"
    )
    use_designer_email = models.BooleanField(
        default=False,
        verbose_name="Use designer's email",
        help_text="Use the designer's email address as the contact email",
    )
    contact_email = models.EmailField(blank=True)

    content_panels = Page.content_panels + [
        FieldPanel("description"),
        MultiFieldPanel(
            [
                FieldPanel("designer"),
                FieldPanel("use_designer_email"),
                FieldPanel("contact_email"),
            ],
            heading="Designer",
        ),
        InlinePanel("campaigns", heading="Campaigns", label="Campaign"),
    ]

    search_fields = Page.search_fields + [
        index.SearchField("description"),
        index.FilterField("designer"),
        index.FilterField("use_designer_email"),
    ]

    @cached_property
    def email(self):
        if (designer := self.designer) and self.use_designer_email:
            return designer.email
        return self.contact_email

Here's what this looks like (Most of the panels have been deliberately collapsed in order to fit everything in one view, for the sake of this illustration) :

Now, here, we want to hide the End date by default, and only display it when the End date is known checkbox is ticked. Generally the same approach we used earlier applies even here. However, we are now dealing with InlinePanels, which introduces some challenges:

1. We have no control over the number of campaigns, which means we have to be very careful about how we select the elements to work with.

2. When the page editor is loaded, there may be no campaigns at all, and additional campaigns can be added at any point.

Once again, the developer tools' DOM inspector is our friend here:

You will notice from the above screenshots that the markup for the end_date inputs in each InlinePanel looks like this (comments added for clarity):

<!-- Campaign 1 -->
<input
  type="text"
  name="campaigns-0-end_date"
  autocomplete="off"
  id="id_campaigns-0-end_date"
/>

<!-- Campaign 2 -->
<input
  type="text"
  name="campaigns-1-end_date"
  autocomplete="off"
  id="id_campaigns-1-end_date"
/>

<!-- Campaign 3 -->
<input
  type="text"
  name="campaigns-2-end_date"
  autocomplete="off"
  id="id_campaigns-2-end_date"
/>

Notice the incremental pattern in the name and id. Similarly, the checkboxes have this markup:

<!-- Campain 1 -->
<input
  type="checkbox"
  name="campaigns-0-end_date_is_known"
  id="id_campaigns-0-end_date_is_known"
/>

<!-- Campain 2 -->
<input
  type="checkbox"
  name="campaigns-1-end_date_is_known"
  id="id_campaigns-1-end_date_is_known"
/>

<!-- Campain 3 -->
<input
  type="checkbox"
  name="campaigns-2-end_date_is_known"
  id="id_campaigns-2-end_date_is_known"
/>

This makes things easier, right? We can see a pattern in the convention used for the input name and id. While this may help us address the first challenge, it may not entirely help us to address the second one. Why? Well, remember we said earlier that

Adding an event listener for the DOMContentLoaded event ensures that the code inside this function runs only when the DOM content is fully loaded and parsed.

When additional campaigns are added after the DOM content is fully loaded and parsed, our code will not work, if we follow the same approach as before. We therefore need to register an event listener on the "add campaign" button:

The button's markup is as follows:

<button
  type="button"
  class="button button-small button-secondary chooser__choose-button"
  id="id_campaigns-ADD"
>
  <svg class="icon icon-plus-inverse icon" aria-hidden="true">
    <use href="#icon-plus-inverse"></use></svg
  >Add campaign
</button>

Alright, talk is cheap, show me the code already! Well, I'm glad you asked, here it is, this time, we'll implement OOP to help keep things more structured:

/**
 * Toggles visibility of an `end_date` field's parent panel based on
 * its counterpart `end_date_is_known` checkbox state'.
 *
 * This is used on the page editor for ToyPages, specifically
 * on the **campaigns** InlinePanel.
 */
class EndDateVisibilityHandler {
  constructor() {
    this.namePrefix = '[name^="campaigns-"]';
    this.idPrefix = '[id^="id_campaigns-"]';
    this.checkboxes = document.querySelectorAll(
      `input[type="checkbox"]${this.namePrefix}${this.idPrefix}[id$="-end_date_is_known"]`
    );
    this.addButton = document.querySelector("#id_campaigns-ADD");
  }

  toggleEndDateField(checkbox) {
    const match = checkbox.id.match(/-(\d+)-end_date_is_known/);
    const identifier = match ? match[1] : null;

    if (identifier !== null) {
      const endDateField = document.getElementById(
        `id_campaigns-${identifier}-end_date`
      );
      const panelWrapper = endDateField.closest(".w-panel__wrapper");
      panelWrapper.style.display = checkbox.checked ? "block" : "none";
    }
  }

  initializeFields(checkboxes) {
    checkboxes.forEach((checkbox) => {
      checkbox.addEventListener("change", () =>
        this.toggleEndDateField(checkbox)
      );
    });
  }

  initializePage() {
    this.initializeFields(this.checkboxes);

    this.addButton.addEventListener("click", () => {
      const newCheckboxes = this.addButton
        .closest(".w-panel__content")
        .querySelectorAll(
          `input[type="checkbox"]${this.namePrefix}${this.idPrefix}[id$="-end_date_is_known"]`
        );
      this.initializeFields(newCheckboxes);
    });
  }
}

document.addEventListener("DOMContentLoaded", () => {
  const edvh = new EndDateVisibilityHandler();
  edvh.initializePage();
});

A few things to note:

1. Notice how we use the attribute selectors [id^=] and [id$=] (Reference) in selecting checkboxes with ids that have an incremental pattern. We know that they have a constant prefix and constant suffix, so we make use of this convention.

2. Once we select a checkbox, we use String.prototype.match() to extract the number from the checkbox's id, which we use to select the accompanying endDateField.

3. When the addButton is clicked, a new InlinePanel is created above the button. Once again, the closest() method comes in handy here.

Here's what this looks like:

Awesome, right? Well, that's all folks! Have fun customizing your Wagtail project!

Please check out the accompanying GitHub repo if you would like to take a closer look and quickly try things out for yourself.

HedgeDoc is an open-source, web-based, self-hosted, collaborative markdown editor. I stumbled upon it while looking for a self-hosted markdown editor. I particularly like it for it's simple and straightforward UI and wide array of features, such as

• diagrams and charts

• media embeds / uploads

• presentations, powered by reveal.js

Dokku is a platform-as-a-service (PaaS) solution that allows you to easily deploy and manage applications on your own infrastructure. It is an open-source alternative to platforms like Heroku, and is designed to be lightweight and straightforward, making it easy for developers to set up and use.

I already have a Dokku instance on an arm64 VPS provided by Hetzner. If you are new to Dokku and would like to give it a spin, please see the official docs. You might also want to have a look at this script I wrote to bootstrap an Ubuntu VPS server prior to installation of Dokku (Use the arm64 branch if you are setting up on an arm64 machine).

I only recently started using servers based on the arm64 architecture, as they have low power consumption and high energy efficiency, which makes them cheaper compared to their x86 counterparts.

Heroku buildpack approach

Dokku has this concept of builders – which are a way of customizing how an app is built from a source. There are several built-in builders available. However, by default, Dokku normally uses Heroku buildpacks (via Herokuish) for deployment.

Now, before I switched to arm64, I had an x86 Dokku server with HedgeDoc and other apps running on it. I had deployed HedgeDoc using the Heroku buildpacks approach, and it worked very well. How did I do this? Well, in a nutshell, I

• forked the https://github.com/hedgedoc/hedgedoc repository to https://github.com/engineervix/hedgedoc

• switched to the master branch, and from there created a dokku branch, where I made some small changes (please see this commit for the details) – addition of a Procfile and installation of pm2, although the latter isn't necessary.

• deployed the dokku branch as follows (all commands are executed on the server, unless otherwise indicated)

    # create app
    dokku apps:create hedgedoc
  
    # add a domain to your app
    dokku domains:add hedgedoc example.com
  
    # setup postgres | https://github.com/dokku/dokku-postgres
    # (feel free to use your preferred database backend)
    dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
    # feel free to choose the image & image version of your choice here
    dokku postgres:create postgres-hedgedoc --image "postgres" --image-version "13.4"
    dokku postgres:link postgres-hedgedoc hedgedoc
    # Take note of the DATABASE_URL, which you will need to assign to CMD_DB_URL later
  
    # set up authentication for backups on the postgres service
    # Datastore backups are supported via AWS S3 and S3 compatible services
    # like https://github.com/minio/minio and https://www.backblaze.com/b2/cloud-storage.html
    # dokku postgres:backup-auth <service> <aws-access-key-id> <aws-secret-access-key> <aws-default-region> <aws-signature-version> <endpoint-url>
    dokku postgres:backup-auth postgres-hedgedoc aws-access-key-id aws-secret-access-key us-west-004 v4 https://s3.us-west-004.backblazeb2.com
    # postgres:backup postgres-hedgedoc <bucket-name>
    dokku postgres:backup postgres-hedgedoc bucket-name
    # everyday at 2:45AM, 10:45AM, 6:45PM
    dokku postgres:backup-schedule postgres-hedgedoc "45 2,10,18 * * *" bucket-name
  
    # Persistent Storage <https://dokku.com/docs/advanced-usage/persistent-storage/>
    # Create a persistent storage directory in the recommended storage path
    dokku storage:ensure-directory --chown heroku hedgedoc
    # Create a new bind mount
    dokku storage:mount hedgedoc /var/lib/dokku/data/storage/hedgedoc/uploads:/home/node/app/public/uploads
  
    # set env variables for your app
    dokku config:set --no-restart hedgedoc CMD_DOMAIN=example.com && \
    dokku config:set --no-restart hedgedoc CMD_ALLOW_ORIGIN='["example.com"]' && \
    dokku config:set --no-restart hedgedoc CMD_SESSION_SECRET=somethingsecret && \
    dokku config:set --no-restart hedgedoc CMD_PROTOCOL_USESSL=true && \
    dokku config:set --no-restart hedgedoc CMD_PORT=3000 && \
    dokku config:set --no-restart hedgedoc CMD_ALLOW_ANONYMOUS=false && \
    dokku config:set --no-restart hedgedoc CMD_ALLOW_EMAIL_REGISTER=false && \
    # NOTE: set CMD_DB_URL to whatever the value of DATABASE_URL is
    dokku config:set hedgedoc CMD_DB_URL=...
  
    # configure buildpacks
    dokku buildpacks:add hedgedoc https://github.com/heroku/heroku-buildpack-nodejs.git
  
    # Customize Nginx | set `client_max_body_size`, to make upload feature work better, for example
    dokku nginx:set hedgedoc client-max-body-size 50m
    # regenerate config
    dokku proxy:build-config hedgedoc
  
    # ports
    dokku ports:add hedgedoc http:80:3000 && \
    dokku ports:add hedgedoc https:443:3000 && \
    dokku ports:remove hedgedoc http:80:5000 && \
    dokku ports:remove hedgedoc https:443:5000
  
    # letsencrypt
    dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
    dokku config:set --no-restart --global DOKKU_LETSENCRYPT_EMAIL=user@example.com
    dokku letsencrypt:set hedgedoc email user@example.com
    dokku letsencrypt:enable hedgedoc
    # this would setup a cron job to renew letsencrypt certificate
    dokku letsencrypt:cron-job --add
  
    # (on local machine) add a remote `dokku` to your git repo, and deploy 🚀
    # git remote add dokku dokku@your-server:hedgedoc
    # git push dokku dokku:master
  
    # after successful deployment, you can add a user
    dokku run hedgedoc bin/manage_users --add user@email.com
  

This worked pretty well, until the fire nation attacked I switched to an arm64 server, where I encountered issues during the deployment, because the Heroku buildpack for Node.js did not work on arm64. I therefore had to consider another approach.

Dockerfile approach

At the time of writing this post, HedgeDoc has a Docker image, which doesn't support the arm64 architecture. However, the good folks at LinuxServer.io have created an Alpine-based multi-arch container image which supports arm64.

I tried two approaches:

1. deploy the linuxserver.io image to my Dokku instance using the Docker Image Deployment approach. I failed to make it work.

2. create my own Dockerfile based on the above image, and deploy via the Dockerfile deployment approach. After many desperate attempts, I still couldn't get things to work.

I ended up giving up. This was in October 2023. The main issue I encountered was that when I ran the application, I couldn't connect to the database, as the application somehow couldn't read the environment variables I had set.

Two months later, I decided to revisit the problem, because I really missed HedgeDoc, having become accustomed to it after previously using it for about a year. I couldn't find any suitable replacement and I was getting frustrated.

I realised that the linuxserver.io image was primarily meant to be deployed via docker compose, so I decided to focus on creating my own Dockerfile, where I would have more control.

This Stack Overflow post was my eureka moment! It seems the environment variables I set (via dokku config:set hedgedoc KEY=VALUE) were not available at build time for non-buildpack-based deploys. According to the Dokku docs:

Environment variables are available both at run time and during the application build/compilation step for buildpack-based deploys.

Even before I stumbled upon the aforementioned Stack Overflow post, I was setting build-time configuration variables (via dokku docker-options:add hedgedoc build '--build-arg KEY=VALUE'), but things were still broken.

What was missing is using both ARG and ENV in the Dockerfile (I was only using ENV), like this:

ARG CMD_DOMAIN
ENV CMD_DOMAIN=${CMD_DOMAIN}

And with this, I was finally able to have a working HedgeDoc deployment 🎉!

So, what does the final code look like? Well, here's a diff between the master branch on the official HedgeDoc repository and the dokku branch on my clone (click on the Files changed tab). The key changes are:

• addition of a Dockerfile

    FROM node:18.18.2-bullseye
  
    RUN mkdir -p /home/node/app && chown -R node:node /home/node/app
  
    # Set the working directory in the container
    WORKDIR /home/node/app
  
    # Port used by this container to serve HTTP.
    EXPOSE 3000
  
    # set environment variables
    # reference: https://docs.hedgedoc.org/configuration/
    ARG CMD_DOMAIN \
        CMD_SESSION_SECRET \
        CMD_ALLOW_ORIGIN \
        CMD_IMAGE_UPLOAD_TYPE \
        CMD_DB_URL
    ENV PORT=3000 \
        CMD_PORT=3000 \
        CMD_IMAGE_UPLOAD_TYPE=${CMD_IMAGE_UPLOAD_TYPE} \
        CMD_ALLOW_ANONYMOUS=false \
        CMD_PROTOCOL_USESSL=true \
        CMD_URL_ADDPORT=false \
        CMD_ALLOW_EMAIL_REGISTER=false \
        CMD_DB_DIALECT=postgres \
        NODE_ENV=production \
        CMD_DOMAIN=${CMD_DOMAIN} \
        CMD_SESSION_SECRET=${CMD_SESSION_SECRET} \
        CMD_ALLOW_ORIGIN=${CMD_ALLOW_ORIGIN} \
        CMD_DB_URL=${CMD_DB_URL} \
        # necessary on ARM because puppeteer doesn't provide a prebuilt binary
        PUPPETEER_SKIP_DOWNLOAD=true
  
    # Yarn 3 requires corepack to be enabled
    RUN corepack enable
  
    # Switch to the non-root user
    USER node
  
    # Copy the code to the container
    COPY --chown=node:node . .
    COPY --chown=node:node config.json.example ./config.json
  
    # Remove the "saml" section from the config.json file
    # because of https://community.hedgedoc.org/t/change-certificate-file-path-of-idp-in-pem-format/108/3
    RUN sed -i '/"saml": {/,/},/d' ./config.json
  
    # Install the application dependencies
    RUN yarn workspaces focus --production
  
    # # Build the frontend bundle
    RUN yarn install --immutable && \
        yarn run build
  
    # Runtime command that executes when "docker run" is called
    # do nothing - exec commands elsewhere
    CMD tail -f /dev/null
  

• addition of a Procfile

    web: yarn start
  

• addition of a .dockerignore file

    .git
    .gitignore
    node_modules
    build
    Dockerfile
    .dockerignore
    .gitignore
    .env
  

As part of the troubleshooting process, I did also modify some files:

• lib/config/default.js

    -  dbURL: '',
    +  dbURL: process.env.DATABASE_URL || '',
  

• lib/models/index.js

      if (config.dbURL) {
    +    logger.info(config.dbURL)
        sequelize = new Sequelize(config.dbURL, dbconfig)
      } else {
    +    logger.warn('config.dbURL is not set')
        sequelize = new Sequelize(dbconfig.database, dbconfig.username, dbconfig.password, dbconfig)
      }
  

I ended up leaving those modifications (I was probably tired and just wanted to use my HedgeDoc!), you don't need to make them, though they might be useful.

Here is how I deployed to Dokku (again, all commands are executed on the server, unless otherwise indicated).

# create app
dokku apps:create hedgedoc

# add a domain to your app
dokku domains:add hedgedoc example.com

# setup postgres | https://github.com/dokku/dokku-postgres
# (feel free to use your preferred database backend)
dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
# feel free to choose the image & image version of your choice here
dokku postgres:create postgres-hedgedoc --image "postgres" --image-version "13.4"
dokku postgres:link postgres-hedgedoc hedgedoc
# Take note of the DATABASE_URL, which you will need to assign to CMD_DB_URL later

# set up authentication for backups on the postgres service
# Datastore backups are supported via AWS S3 and S3 compatible services
# like https://github.com/minio/minio and https://www.backblaze.com/b2/cloud-storage.html
# dokku postgres:backup-auth <service> <aws-access-key-id> <aws-secret-access-key> <aws-default-region> <aws-signature-version> <endpoint-url>
dokku postgres:backup-auth postgres-hedgedoc aws-access-key-id aws-secret-access-key us-west-004 v4 https://s3.us-west-004.backblazeb2.com
# postgres:backup postgres-hedgedoc <bucket-name>
dokku postgres:backup postgres-hedgedoc bucket-name
# everyday at 2:45AM, 10:45AM, 6:45PM
dokku postgres:backup-schedule postgres-hedgedoc "45 2,10,18 * * *" bucket-name

# Persistent Storage <https://dokku.com/docs/advanced-usage/persistent-storage/>
# Create a persistent storage directory in the recommended storage path
dokku storage:ensure-directory --chown heroku hedgedoc
# Create a new bind mount
dokku storage:mount hedgedoc /var/lib/dokku/data/storage/hedgedoc/uploads:/home/node/app/public/uploads

# set env variables for your app
# I wonder whether this is even necessary, since we are actually using Docker build args
dokku config:set --no-restart hedgedoc CMD_DOMAIN=example.com && \
dokku config:set --no-restart hedgedoc CMD_ALLOW_ORIGIN='["example.com"]' && \
dokku config:set --no-restart hedgedoc CMD_SESSION_SECRET=somethingsecret && \
dokku config:set --no-restart hedgedoc CMD_IMAGE_UPLOAD_TYPE=filesystem && \
# NOTE: set CMD_DB_URL to whatever the value of DATABASE_URL is
dokku config:set hedgedoc CMD_DB_URL=...

# customize Docker Build-time configuration variables
# https://dokku.com/docs/deployment/builders/dockerfiles/#build-time-configuration-variables
# --------------------------------------------------------------------------------------------
dokku docker-options:add hedgedoc build '--build-arg CMD_DOMAIN=example.com' && \
dokku docker-options:add hedgedoc build '--build-arg CMD_ALLOW_ORIGIN=["example.com"]' && \
dokku docker-options:add hedgedoc build '--build-arg CMD_SESSION_SECRET=somethingsecret' && \
dokku docker-options:add hedgedoc build '--build-arg CMD_IMAGE_UPLOAD_TYPE=filesystem' && \
dokku docker-options:add hedgedoc build '--build-arg CMD_DB_URL=postgres://username:password@host:port/database'
# add others based on your requirements. See https://docs.hedgedoc.org/configuration
# NOTE: I just saw [from the docs](https://dokku.com/docs/deployment/builders/dockerfiles/#build-time-configuration-variables) as I was writing this, that:
# "All environment variables set by the config plugin are automatically exported during a docker build,
#  and thus --build-arg only requires setting a key without a value.
#
#  dokku docker-options:add node-js-app build '--build-arg NODE_ENV'
# "

# Customize Nginx | set `client_max_body_size`, to make upload feature work better, for example
dokku nginx:set hedgedoc client-max-body-size 50m
# regenerate config
dokku proxy:build-config hedgedoc

# ports
dokku ports:add hedgedoc http:80:3000 && \
dokku ports:add hedgedoc https:443:3000 && \
dokku ports:remove hedgedoc http:80:5000 && \
dokku ports:remove hedgedoc https:443:5000

# letsencrypt
dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
dokku config:set --no-restart --global DOKKU_LETSENCRYPT_EMAIL=user@example.com
dokku letsencrypt:set hedgedoc email user@example.com
dokku letsencrypt:enable hedgedoc
# this would setup a cron job to renew letsencrypt certificate
dokku letsencrypt:cron-job --add

# (on local machine) add a remote `dokku` to your git repo, and deploy 🚀
# git remote add dokku dokku@your-server:hedgedoc
# git push dokku dokku:master

# after successful deployment, you can add a user
dokku run hedgedoc bin/manage_users --add user@email.com

You will observe that the commands executed above are not so different from the ones for the Heroku buildpack approach. The main differences:

Well, I hope you find this useful. If you have any suggestions, questions or comments, please feel free to comment below. Enjoy your HedgeDoc!

So, without any knowledge and experience in building things with LLMs, I decided to embark on a quest to build a tool that automatically gathers news from various sources in my country 🇿🇲, summarizes the news articles and produces a podcast, without my input!

TLDR; here's the result, and here's the code.

The rationale

When starting this project, I already knew from the Hackercast project that I needed to

• scrape various news websites,

• use Langchain to summarize the news articles and

• use AWS Polly to convert text to speech.

I was already familiar with using requests and Beautiful Soup for scraping, but I had to learn how to use Langchain and AWS Polly. I tried to consider alternatives to AWS Polly but after doing some bit of searching and experimentation, I concluded that AWS Polly was the best tool for the job, because of its

• reasonable pricing

• nice, straightforward API via Boto3.

• many language options and features

• nice documentation

• excellent integration with S3, which means that I have fewer headaches about where to store the audio files.

However, scraping, summarization and text-to-speech were only a small piece of the puzzle. I had to think about how the whole thing was going to work. Remember, the problem I'm trying to solve here is staying up to date with the latest news. I do not have time to scour through news websites and read the news, so this tool I'm building should do the job, and tell me what's going on! From the onset, I wanted

• the news to be automatically delivered every weekday in the afternoon, towards the end of my work day, so I can listen to the news on my commute or something along those lines. This has cron written all over it!

• the news to be accessible from anywhere, on multiple podcast platforms. This meant I had to build my own web application with HTML Audio and a proper RSS feed which would be submitted to the various platforms.

AWS Polly, S3 and OpenAI are not free. It didn't make sense to build a web application that would require additional expenses for hosting, since the main content would be audio files, which would be on an S3 bucket. I therefore concluded that I would have to build a static site and deploy it for free via services like Github Pages, Cloudflare Pages, render, etc.

Now, which static site generator do I use? This being a Python project, I could have used the likes of Pelican or even Frozen Flask. However, it's been a while since I played around with Pelican, and I've never even used Frozen Flask. Besides, using Flask would be overkill for this. I was already learning so many things, and didn't want to spend too much time tinkering with shiny new toys, at the expense of actually solving the problem at hand and getting things done. So I resorted to using 11ty, which I was very familiar with and had used recently. I particularly love 11ty as I find it simple to use, yet very powerful and unopinionated, which means I can build anything with it, with the freedom to choose which template engine to use, how to structure my project, which build tools to use, etc.

To sum it up, I broke down the project into two main components:

• the core – which consists of the Python code that handles the fetching of news items, summarization, audio production and creation of content to be passed to the web app.

• the web app – which renders the generated content.

Now, you may ask, "What's with the name?". Well, I chose zed-news because Zed is for Zambia, and this whole project is a podcast consisting of news curated from Zambian sources. That makes sense, right? Well, naming things is hard!

The experience

Building zed-news was an interesting and fun challenge. I learnt a lot of things along the way, and I had to rewrite some things and make some important implementation decisions. I will highlight a few noteworthy aspects here.

Making things less "robotic" and more "natural"

The result of this project is an audio file consisting of the news read by an AWS Polly voice. To minimise monotony and avoid having each episode sound the same,

• I grouped several phrases/sentences and picked a random one at various points when constructing the transcript. Here's an example

    import random
  
    def random_dig_in():
        variations = [
            "Without any more delay, let's jump right in.",
            "No time to waste, let's get started.",
            "Let's not wait any longer, it's time to delve in.",
            "Without further ado, let's dive in.",
            "Without prolonging the anticipation, let's begin our exploration.",
            "Time to embark on our news journey. Let's get to it.",
        ]
  
        return random.choice(variations)
  

• I used the num2words library to convert numbers to words. This was particularly useful for generating ordinal numbers like seventh or thirty-first. Here's an example of how I used it

    import random
    from num2words import num2words
  
    read = ""
    # ...
    # ...
    # Iterate over each article in the source
    for index, article in enumerate(articles_by_source[source], start=1):
        count = num2words(index)
        count_ordinal = num2words(index, to="ordinal")
        count_variations = [
            f"Entry number {count} ",
            f"The {count_ordinal} entry ",
        ]
        title = article["title"]
        read += f"{random.choice(count_variations)} is entitled '{title}' "
  

Keeping track of the episode number

Because I wanted to have this project run on auto-pilot, I needed a way for it to know that today's episode is number 8, for example. Initially, I thought I was going to run this every day, so I based the episode number on the first date the podcast was published. However, I quickly saw that this would be a problem because anything could go wrong and no podcasts would be published on certain days. Besides, I later decided to just run this on weekdays. Therefore, I needed to keep track of the episode number so that I could just increment when publishing a new episode.

Serverless Postgres with Neon

I decided to not only keep track of the episode number but also all the pertinent episode information, and thought a database would be a good solution. I have always wanted to try out the Serverless Postgres offering by neon.tech, so this was a good opportunity to do so. The whole process has been pretty smooth so far.

Choosing an ORM was difficult, I ended up settling for Tortoise ORM because I'd previously played around with it while learning FastAPI. The main reason I liked Tortoise ORM is because it has a syntax similar to the Django ORM, which means a lower learning curve for me, or so I thought ...

Async / Await

... You see, Tortoise ORM is an asyncio ORM, which introduces additional complexities into the mix (See this article by Charles Leifer, and this one by Armin Ronacher). I spent a lot of time fixing async / await issues, and I still don't really know what's going on under the hood! But hey, it works, right?😀 Well, that's a subject for another day!

Summarization

It wasn't hard to get up and running with Langchain, which provides a very nice API to work with LLMs. The resource "5 Levels Of Summarization: Novice to Expert" was very helpful. I went with the simplest approach, as it was sufficient for this project.

I tried playing around with Cohere's Summarize API, but I found it a bit disappointing because of crazy hallucinations where it mixed up content based on old events.

One issue I encountered was exceeding OpenAI's token limit of 4096 tokens. This happened when pricing longer articles. The typical solution for this is to split the content into smaller chunks that fit within the token limit, summarize each chunk, and then get a summary of the summaries. The technical term is referred to as Map - Reduce, and falls under Level 3 in the above resource. However, I didn't bother to go for this technique, because I didn't have the time. So I just wrote a quick and dirty hack, where I truncate the content to fit within the token limit and ignore the rest of the content:

import logging
import math
import textwrap

from langchain import OpenAI, PromptTemplate

from app.core.utilities import OPENAI_API_KEY

llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)
MAX_TOKENS = 4096


def summarize(content: str, title: str) -> str:
    """Summarize the content using OpenAI's language model."""

    template = """
    Please provide a very short, sweet, informative and engaging summary of the following news entry, in not more than two sentences.
    Please provide your output in a manner suitable for reading as part of a podcast.

    {entry}
    """

    # Calculate the maximum number of tokens available for the prompt
    max_prompt_tokens = MAX_TOKENS - llm.get_num_tokens(template)

    # Trim the content if it exceeds the available tokens
    # TODO: Instead of truncating the content, split it
    # see <https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/split_by_token>
    chars = int(max_prompt_tokens * 3.75)  # Assuming 1 token ≈ 4 chars
    # round down max_chars to the nearest 100
    max_chars = math.floor(chars / 100) * 100
    if len(content) > max_chars:
        content = textwrap.shorten(content, width=max_chars, placeholder=" ...")

    prompt = PromptTemplate(input_variables=["entry"], template=template)

    summary_prompt = prompt.format(entry=content)

    num_tokens = llm.get_num_tokens(summary_prompt)
    logging.info(f"'{title}' and its prompt has {num_tokens} tokens")

    return llm(summary_prompt)

Defining the toolchain and cron job

I'm a big fan of Invoke, which I wrote about in this post. After coding all the bits and pieces, I tied everything together with Invoke. Well, I actually just created a run.py script with a main() function that runs when the run.py script is executed directly. Then I created an Invoke task:

@task
def toolchain(c):
    """The toolchain for creating the podcast"""
    c.run("python app/core/run.py", pty=True)

The idea was to run inv toolchain as part of a cron job script on an Ubuntu VPS. I wrote a cron.sh BASH script and added a bunch of commands there. I made use of healthchecks.io to ensure that I'm aware when the script fails to run. Here's what the script looks like:

#!/usr/bin/env bash

# ==========================================================
# Logical steps
# 1. cd to project directory
# 2. Activate virtual environment
# 3. git pull
# 4. Run script inside docker container (cleanup afterwards)
# 5. commit changes
# 6. push changes to remote
# ==========================================================

set -e  # Exit immediately if any command fails

# 1. cd to project directory
cd "${HOME}/SITES/zed-news" || { echo "Failed to change directory."; exit 1; }

# Source the .env file so we can retrieve healthchecks.io ping URL
# shellcheck source=/dev/null
source .env

# Function to send success signal to healthchecks.io
function send_healthcheck_success() {
  curl -fsS --retry 3 "${HEALTHCHECKS_PING_URL}" > /dev/null
}

# Function to send failure signal to healthchecks.io
function send_healthcheck_failure() {
  curl -fsS --retry 3 "${HEALTHCHECKS_PING_URL}/fail" > /dev/null
}

# 2. Activate virtual environment
# shellcheck source=/dev/null
source "${HOME}/Env/zed-news/bin/activate" || { echo "Failed to activate virtual environment."; send_healthcheck_failure; exit 1; }

# 3. git pull
git pull || { echo "Failed to pull changes from Git."; send_healthcheck_failure; exit 1; }

# 4. Run script inside docker container
inv up --build || { echo "Failed to build Docker container."; send_healthcheck_failure; exit 1; }
docker-compose run --rm app invoke toolchain || { echo "Failed to run script inside Docker container."; send_healthcheck_failure; exit 1; }
inv down || { echo "Failed to stop Docker container."; send_healthcheck_failure; exit 1; }

# 5. commit changes
today_iso=$(date --iso)
git add . || { echo "Failed to stage changes for commit."; send_healthcheck_failure; exit 1; }
git commit --no-verify -m "chore: ✨ new episode 🎙️ » ${today_iso}" || { echo "Failed to commit changes."; send_healthcheck_failure; exit 1; }

# 6. push changes to remote
git push origin main || { echo "Failed to push changes to remote repository."; send_healthcheck_failure; exit 1; }

# Send success signal to healthchecks.io
send_healthcheck_success

# Pause for 5 minutes
sleep 300

# Notify Admin via Telegram
# See next section ...

Pushing the changes to remote triggers a Cloudflare Pages build & deployment. I pause the script for 5 minutes to allow this process to complete, before notifying myself.

Notifying myself when a new episode is published

As I have already mentioned, this project was really a fantastic learning opportunity for me. I learned how to use apprise, by configuring it to send myself a message via Telegram when a new episode was published. This is the last step in the cron.sh script:

# Notify Admin via Telegram
today_human_readable=$(date +"%a %d %b %Y")
apprise -vv -t "🎙️ New Episode » ${today_human_readable}" \
  -b "📻 Listen now at ${BASE_URL}/episode/${today_iso}/" \
  tgram://"${TELEGRAM_BOT_TOKEN}"

Switching from pip-tools to poetry

I have been using pip-tools in my projects for a while, I even wrote about it here. However, I use Poetry a lot at work, and have come to love it! Initially, I just wanted to update my pip-tools setup to use pyproject.toml to manage dependencies, instead of multiple requirements* files. However, I ended up convincing myself to just switch to Poetry, which I did, and it was fun porting from pip-tools!

Switching from flake8 / isort/ pycodestyle to ruff

Having heard about ruff, I decided to give it a try on this project. One of the motivations for this was that I would have fewer config files in my project root since ruff uses pyproject.toml! Also, it's one linting command instead of several! The switch was fairly smooth, I didn't encounter any issues that I can immediately recall. My Invoke linting task looks like this:

@task(help={"fix": "let black and ruff format your files"})
def lint(c, fix=False):
    """ruff and black"""

    if fix:
        c.run("black .", pty=True)
        c.run("ruff check --fix .", pty=True)
    else:
        c.run("black . --check", pty=True)
        c.run("ruff check .", pty=True)

And the GitHub Actions job looks like this

jobs:
  linter_ruff:
    runs-on: ubuntu-22.04
    container: python:3.10-slim-bullseye

    steps:
      - name: Checkout Code Repository
        uses: actions/checkout@v3

      - name: Install Dependencies
        run: |
          pip install -q ruff==0.0.270
      - name: ruff
        run: |
          ruff check --format=github .

Audio mixing on auto-pilot

In December 2022, I wrote about using CLI tools to manage audio, video, images and PDF files. One of these tools is FFmpeg, and it came in quite handy in this project because I had to automate various aspects of the audio production, as follows:

1. Adjustment of the AWS Polly generated audio file's sample rate from 24 kHz to 44.1 kHz. This is the standard for most consumer audio and is used for formats like CDs.

2. Converting the result from (1) above from mono to 128 kb/s stereo.

3. Adjustment of the treble (high frequency).

4. Addition of both an intro and outro instrumental.

5. Addition of ID3 tags to the final audio file

I wrote a function to do these things:

# For more details, see 
# https://github.com/engineervix/zed-news/blob/v0.3.1/app/core/podcast/mix.py

async def mix_audio(voice_track, intro_track, outro_track, dest=f"{DATA_DIR}/{today_iso_fmt}_podcast_dist.mp3"):
    """
    Mix the voice track, intro track, and outro track into a single audio file
    """

    voice_track_file_name = os.path.splitext(voice_track)[0]
    mix_44100 = f"{voice_track_file_name}.44.1kHz.mp3"
    voice_track_in_stereo = f"{voice_track_file_name}.stereo.mp3"
    eq_mix = f"{voice_track_file_name}.eq-mix.mp3"
    initial_mix = f"{voice_track_file_name}.mix-01.mp3"

    # change the voice track sample rate to 44.1 kHz
    subprocess.run(
        f"ffmpeg -i {voice_track} -ar 44100 {mix_44100}",
        shell=True,
    )

    # convert voice track from mono to 128 kb/s stereo
    subprocess.run(
        f'ffmpeg -i {mix_44100} -af "pan=stereo|c0=c0|c1=c0" -b:a 128k {voice_track_in_stereo}',
        shell=True,
    )

    # adjust the treble (high-frequency).
    # The g=3 parameter specifies the gain in decibels (dB) to be applied to the treble frequencies.
    subprocess.run(
        f'ffmpeg -i {voice_track_in_stereo} -af "treble=g=3" {eq_mix}',
        shell=True,
    )

    # initial mix: the intro + voice track
    subprocess.run(
        f'ffmpeg -i {eq_mix} -i {intro_track} -filter_complex amix=inputs=2:duration=longest:dropout_transition=0:weights="1 0.25":normalize=0 {initial_mix}',
        shell=True,
    )

    # get duration of the initial mix
    command_1 = f'ffmpeg -i {initial_mix} 2>&1 | grep "Duration"'
    output_1 = run_ffmpeg_command(command_1)
    duration_1 = extract_duration_in_milliseconds(output_1)

    # get duration of the outro instrumental
    command_2 = f'ffmpeg -i {outro_track} 2>&1 | grep "Duration"'
    output_2 = run_ffmpeg_command(command_2)
    duration_2 = extract_duration_in_milliseconds(output_2)

    # pad the outro instrumental with silence, using initial mix duration and
    # the outro instrumental's duration
    # adelay = (duration of initial mix - outro instrumental duration) in milliseconds
    if duration_1 != 0 and duration_2 != 0:
        padded_outro = f"{voice_track_file_name}.mix-02.mp3"

        adelay = duration_1 - duration_2
        subprocess.run(f'ffmpeg -i {outro_track} -af "adelay={adelay}|{adelay}" {padded_outro}', shell=True)

        # final mix: the initial mix + the padded outro
        subprocess.run(
            f'ffmpeg -i {initial_mix} -i {padded_outro} -filter_complex amix=inputs=2:duration=longest:dropout_transition=0:weights="1 0.25":normalize=0 {dest}',
            shell=True,
        )

        # add Id3 tags
        episode = await get_episode_number()
        audio_file = dest
        tag = eyed3.load(audio_file).tag
        tag.artist = "Victor Miti"
        tag.album = "Zed News"
        tag.title = f"Zed News Podcast, Episode {episode:03} ({today_human_readable})"
        tag.track_num = episode
        tag.release_date = eyed3.core.Date(today.year, today.month, today.day)
        tag.genre = "Podcast"
        album_art_file = f"{IMAGE_DIR}/album-art.jpg"
        with open(album_art_file, "rb") as cover_art:
            # The value 3 indicates that the front cover shall be set
            # # https://eyed3.readthedocs.io/en/latest/eyed3.id3.html#eyed3.id3.frames.ImageFrame
            tag.images.set(3, cover_art.read(), "image/jpeg")
        tag.save()

        # Clean up
        for f in [voice_track_in_stereo, mix_44100, eq_mix, initial_mix, padded_outro]:
            delete_file(f)

The RSS Feed

People typically use podcast management tools such as SimpleCast, Buszzsprout, Podbean, etc. These manage various aspects of the podcast publishing process, including file hosting, and submission to Google Podcasts, Apple Podcasts and all the other major podcast platforms. In my case, I was essentially rolling out my own solution, which meant that I had to deal with all these tasks myself. I had already sorted out the file hosting by using AWS S3. The main thing I needed to do next was automatic submission of my podcast to various platforms. It turns out that the key to doing this properly is having a well-structured RSS Feed with the right metadata. I found the following resources to be very helpful:

• A Podcaster's Guide to RSS, by Apple

• RSS feed guidelines for Google Podcasts, by Google

Getting this part of the project right involved a bit of trial and error, this is the Nunjucks template I ended up with for generating the feed.xml file:

---json
{
  "permalink": "feed.xml",
  "eleventyExcludeFromCollections": true,
  "metadata": {
    "title": "Zed News Podcast",
    "description": "Your weekday source for the latest happenings in Zambia (and beyond), the Zed News Podcast is an automated news podcast consisting of AI-powered updates from various Zambian sources. New episodes Monday to Friday between 16:45 and 17:00 CAT.",
    "url": "https://example.com",
    "feedUrl": "https://example.com/feed.xml",
    "author": {
      "name": "Victor Miti",
      "email": "victormiti@gmail.com"
    }
  }
}
---
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
     xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
     xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>{{ metadata.title }}</title>
        <description>{{ metadata.description }}</description>
        <itunes:owner>
            <itunes:name>{{ metadata.author.name }}</itunes:name>
            <itunes:email>{{ metadata.author.email }}</itunes:email>
        </itunes:owner>
        <itunes:author>{{ metadata.author.name }}</itunes:author>
        <itunes:image href="{{ site.base_url }}/img/zed-news-podcast-album-art.jpg"/>
        <itunes:category text="News &amp; Politics" />
        <itunes:explicit>no</itunes:explicit>
        <language>en</language>
        <link href="{{ site.base_url }}" rel="self"/>
        <link href="{{ site.base_url }}"/>
        {%- for episode in collections.episode | reverse -%}
        {% set absolutePostUrl %}{{ episode.url | url | absoluteUrl(site.base_url) }}{% endset %}
        <item>
            <title>{{ episode.data.title }}</title>
            <description>{{ episode.data.description }} It is presented by {{ episode.data.presenter }} in {{ episode.data.locale.name }} and consists of {{ episode.data.references.count }} articles from {{ episode.data.references.sources }} sources. The size of the audio file is {{ episode.data.mp3.size }}. Running time is {{ episode.data.mp3.length }}.Check out the <![CDATA[ <a href="{{ absolutePostUrl }}">episode notes</a> ]]> for more information, including links to the news sources.</description>
            <pubDate>{{ episode.date | dateToRfc822 }}</pubDate>
            <enclosure url="{{ episode.data.mp3.url }}"
                       type="audio/mpeg" length="{{ episode.data.rss.enclosure_length }}"/>
            <itunes:duration>{{ episode.data.rss.itunes_duration }}</itunes:duration>
            <link>{{ absolutePostUrl }}</link>
            <guid>{{ absolutePostUrl }}</guid>
            <itunes:explicit>no</itunes:explicit>
        </item>
        {%- endfor %}
    </channel>
</rss>

With this in place, I just submitted https://zednews.pages.dev/feed.xml to the relevant platforms and that was it.

Rendering a different button depending on OS

I wanted to show a "Listen on Apple Podcasts" button when the site was accessed from an Apple device, and "Listen on Google Podcasts" for all other devices. I used UAParser.js to achieve this:

function generatePodcastListenButton() {
  const parser = new UAParser();
  const result = parser.getResult();

  const platform = result.os.name.toLowerCase();
  let buttonHTML = "";

  if (platform === "mac os" || platform === "ios") {
    buttonHTML = `
      <a class="podcast-listen-btn" href="https://podcasts.apple.com/us/podcast/zed-news-podcast/id1690709989">
        <img src="/img/apple-podcasts-badge.svg" alt="Listen on Apple Podcasts" height="40" />
      </a>
    `;
  } else {
    buttonHTML = `
      <a class="podcast-listen-btn" href="https://podcasts.google.com/feed/aHR0cHM6Ly96ZWRuZXdzLnBhZ2VzLmRldi9mZWVkLnhtbA">
        <img src="/img/google-podcasts-badge.svg" alt="Listen on Google Podcasts" height="38" />
      </a>
    `;
  }

  return buttonHTML;
}

Writing unit tests without using pytest

I am used to writing tests using pytest. However, I decided to challenge myself to improve my knowledge and understanding of unittest, which is part of the standard library. I recently came to know about the left-pad chaos, and was challenged by this thought-provoking article. I already have too many dependencies on this project, and I didn't want to install more stuff and start configuring pytest, when I could achieve the same thing with unittest and even learn one thing or two in the process. I did spend quite some time writing tests, and I must say I enjoyed the process and gained valuable knowledge and experience, particularly regarding mocking.

Cloudflare Pages Issues

I use Cloudflare for my DNS setups. I prefer Cloudflare Pages for deploying static sites because it is part of the Cloudflare suite of tools, which makes it easy for me to manage everything in one place. However, at the time of deploying the project, Cloudflare Pages' v1 build system (stable) supported old versions of various software. For example, the default Python version is 2.7, and only Python versions 2.7, 3.5 and 3.7 were supported. I didn't think this would affect me, because the build command for the generated static site did not need Python at all. However, I was wrong! The build system looks for certain files in the project root and makes assumptions based on those files. In my case, it "knew" this was a Python project and therefore attempted to install the project dependencies. This failed because the project uses Python 3.10.

I therefore had to change the build settings to use the v2 build system (beta). The build process still failed, because it couldn't install dependencies using poetry! So I had to generate a requirements.txt file from poetry:

poetry export --with dev --without-hashes --format requirements.txt --output requirements.txt

The other issue, which I actually encountered on a different project, had to do with the Node version. I experienced build failures using Node.js v18. I only fixed the problem by downgrading to v16. With this in mind, I maintained the project's Node.js version to v16, as I wasn't prepared to experiment with v18 again. Perhaps this may have been fixed by now, I'll need to check at some point.

The result

Well, here it is ...

On Mobile

On Desktop

It's not perfect, but it works! Check it out and let me know what you think. The code is open, feel free to add your contributions to make it better!

Cover image generated by Bing.

1. I never thought I'd be switching careers and becoming a developer

2. I met and worked with so many new people

3. I learnt a lot of new things and unlearned some old ways of doing things

4. I presented a talk at a tech conference for the first time

5. ... I could go on and on ... but, let's keep it simple 😃

Now, let's break things down a little bit 🙂

Getting my first role as a developer

In March 2022, I started work as a junior developer at Torchbox, working remotely and building interesting stuff with Python, Django and Wagtail! I'm grateful to The Lord for this opportunity. I never imagined myself leaving my previous job in 2022 to become a developer. God willing, I will write about this in a separate post. Suffice it to say, it's not easy to leave behind a career you've built for a decade, to start afresh in a completely new field. Notwithstanding, it's been a great experience so far, and I hope I can continue to grow in this new craft, and become a better developer one step at a time 🏋💪!

In the meantime, check out our careers page, you might just be my teammate! There are various roles there, across various departments.

New people, new ways of working

Starting a new job not only entails meeting and working with new people, but also new ways of working! In my case, all my colleagues are at least 10,000km away, on various continents outside Africa, and I haven't even met any of them in person! However, they are all fantastic people, and it has been such a joy working and interacting with them. I thought working remotely would be all fun and games ...

It requires a lot of discipline, focus and hard work, otherwise you won't get anything done!

The whole agile thing was new to me — sprints, stand-ups, backlog refinements, etc. Took a while to get used to this, and I'm still learning!

Prior to this experience, most of the development work I did was solo. No teams, no collaborators, except, of course, for open source contributions. Now, I learnt all about Git workflows, and the various nuances of having several pieces of interdependent work going on simultaneously ... interesting stuff!

All in all, a lot of adjustments and adaptation, but all good so far 👍

Learning, learning and learning

I've already highlighted some of the things I've had to learn. I think there's been so much learning in 2022 for me. At the end of the year, I was certainly a better developer than I was at the beginning of 2022. I even subscribed to Coursera Plus at the end of the year, so that I could continue to level up my skills! I started a Computer Science Series, where I talk about these things. You'll also see from my Class Central profile that I completed the following two courses:

• Fundamentals of Digital Marketing, by Google.

• Starting Up, by Aalto University and MinnaLearn.

"But these courses don't even have anything to do with programming", you might say! Well, that isn't entirely correct! Being a developer is much more than writing good code. Ask John Sonmez, he has a lot to say about this! I have been reading his book and I have found it to be quite insightful.

First PyCon talk

I submitted a last-minute application to present a remote talk at PyCon ZA 2022, and I couldn't believe it when I got the email confirmation of acceptance! You can check out my talk in the video below:

And that's a wrap!

Well, I'll pause here for now. I need to get some sleep 😴, a busy week ahead, lots of things to do, and more interesting things to learn! By God's grace, we keep moving!

• FFmpeg: A complete, cross-platform solution to record, convert and stream audio and video

• Gifsicle: for creating, editing, and getting information about GIF images and animations

• ImageMagick: create, edit, compose, or convert digital images

• PDFtk: for doing everyday things with PDF documents.

• OCRmyPDF: adds an OCR text layer to scanned PDF files, allowing them to be searched

• ghostscript: an interpreter for PostScript® and Portable Document Format (PDF) files.

• mozjpeg: improves JPEG compression efficiency achieving higher visual quality and smaller file sizes at the same time

• pngquant: a utility and library for lossy compression of PNG images. The conversion reduces file sizes significantly (often as much as 70%) and preserves full alpha transparency.

• svgo: a Node.js-based tool for optimizing SVG vector graphics files.

This is a work in progress. I'll be updating the commands from time to time, as I encounter them.

FFmpeg

convert video from x to y

Generally speaking, you should be able to convert a video from one format to another by using the following convention:

# x and y here are the file extensions, for instance; mp4, mkv, ogv
ffmpeg -i input.x -c copy output.y

If you encounter any issues, a quick Google search for How to convert x to y using FFmpeg will give you the solution.

Reference(s)

• webm to mp4 conversion using ffmpeg

compress video (reduce video size, without a noticeable reduction in quality)

Here, I'm using mp4 videos. If you have other video types (mkv, ogv, webm, etc. then you might need to convert those to mp4 first

# using the libx265 codec
ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4

# if you don't have libx265, you can use the libx264 codec
ffmpeg -i input.mp4 -vcodec libx264 -crf 28 output.mp4

For more info & technical details, see this Unix & Linux Stack Exchange post: How can I reduce a video's size with ffmpeg?

speed up a video

I find this useful in situations where I record a screen capture (no audio) while doing a demo, and I end up with, say, a 5-minute video that has several periods of no activity (e.g. waiting for a process to complete). I could speed up the video and hence reduce its overall time to, say 2.5 minutes:

# the 0.5 here means we're reducing the video to half the original time.
# adjust this to suit your preference. If the value is above 1.0, then you're making your video slower
ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" output.mp4

If you wanna speed up video and audio at the same time:

# note here that the audio tempo is the inverse of the factor modifying frame timestamps. In English,
# if you're speeding up the video 2x, divide 1 by 2 to get 0.5 for the video, then
# use 2.0 as the factor for the audio tempo (this is 1 divided by video factor)
ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" output.mp4

For more info and technical details, see

• How to speed up / slow down a video on the FFmpeg Bug Tracker and Wiki.

• Speed up video x1.5 but keep all frames on superuser.com.

concatenate video / audio

The assumption here is that you want to concatenate files with the same codecs. If this isn't the case, see this page on the FFmpeg Bug Tracker and Wiki.

What you wanna do is create a text file with all the files (whose paths can be either relative or absolute) you want to have concatenated in the following form:

file '/path/to/file1.mp3'
file '/path/to/file2.mp3'
file '/path/to/file3.mp3'

It is possible to generate this file with a bash for loop, or using printf. Either of the following would generate a file containing every *.mp3 in the working directory:

# with a bash for loop
for f in *.mp3; do echo "file '$f'" >> text_file.txt; done

# or with printf
printf "file '%s'\n" *.mp3 > mylist.txt

Then, using FFmpeg:

# The -safe 0 is not required if the paths are relative
ffmpeg -f concat -safe 0 -i text_file.txt -c copy output.mp3

For more info and technical details, see

• Concatenating media files on Stackoverflow

• How to concatenate two MP4 files using FFmpeg? on the FFmpeg Bug Tracker and Wiki

extract audio from video

If we don't wanna re-encode, then we first have to run ffprobe on the video file to list available audio streams and their types:

❯ ffprobe video.mp4
ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12 (SUSE Linux)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --optflags='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -ffat-lto-objects -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --enable-version3 --disable-openssl --enable-gnutls --enable-ladspa --enable-libshaderc --enable-vulkan --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdc1394 --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libjack --enable-libjxl --enable-librist --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopenh264-dlopen --enable-libopus --enable-libpulse --enable-librav1e --enable-librubberband --enable-libsvtav1 --enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lto --enable-lv2 --enable-libmfx --enable-vaapi --enable-vdpau --enable-version3 --enable-libfdk-aac-dlopen --enable-nonfree --enable-libvo-amrwbenc --enable-libx264 --enable-libx265 --enable-libxvid
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:02:32.96, start: 0.000000, bitrate: 452 kb/s
  Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 126 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720, 317 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]

From above, we see that the audio stream format is aac. Then we can go ahead and

ffmpeg -i video.mp4 -map 0:a -acodec copy audio.aac

However, if we want to extract the audio in a specific format, say mp3, then we'll have to re-encode:

# here we're using libmp3lame <https://www.ffmpeg.org/ffmpeg-codecs.html#libmp3lame-1> for mp3 encoding
ffmpeg -i video.mp4 -map 0:a -acodec libmp3lame audio.mp3

compress audio

You have an hour-long 320kbps mp3 file weighing in at 144Mb. Let's say you want to reduce the target bitrate to 96kbps:

# -q:a 7 gives you an average bitrate of 100kbps
# for reference, see <https://trac.ffmpeg.org/wiki/Encode/MP3>
ffmpeg -i input.mp3 -codec:a libmp3lame -q:a 7 output.mp3

That should give you a resulting file ~45Mb! You'll have to play the file to determine if you are happy with the quality. Note that audio quality improves with increasing bitrate. Here's a quick guide (taken from wikipedia.org/wiki/Bit_rate#MP3):

• 32 kbps – generally acceptable only for speech

• 96kbps – generally used for speech or low-quality streaming

• 128 or 160 kbps – mid-range bitrate quality

• 192 kbps – medium-quality bitrate

• 256 kbps – a commonly used high-quality bitrate

• 320 kbps – highest level supported by the MP3 standard

Note(s)

• For additional compression, you could mix down to mono (use 1 audio channel) and set the sample rate to 22050 Hz (which is half the standard 44.1kHz) by using -ac 1 for the former and -ar 22050 for the latter.

• Refer to the FFmpeg MP3 Encoding Guide for more info and technical details

• Also, check out Manipulating audio channels on the FFmpeg Bug Tracker and Wiki

extract part of a media file

Say you have a 45min audio file and you only need the part between 2min and 44min:

# time format is HH:MM:SS or HH:MM:SS.xx, where xx expresses decimal value for SS
# see <https://ffmpeg.org/ffmpeg-utils.html#Time-duration>
ffmpeg -i input.mp3 -ss 00:02:00 -to 00:44:00.00 [other options here] output.mp3

Gifsicle

compress a GIF file

gifsicle -O3 --colors 256 --lossy=30 -o output.gif input.gif

Reference: Optimize animated GIF size in command-line on superuser.com

Imagemagick

resize images

Using image width as a reference

# we want to resize the image, keeping the width at 1280px
convert -resize 1280x input.jpg output.jpg

Using image height as a reference

# we want to resize the image, keeping the height at 1280px
convert -resize x1280 input.jpg output.jpg

convert X to Y

Convert, say, a JPG file to PNG

convert image.png image.jpg

Convert PNG with transparency to JPG

# -flatten, by default, results in a white background
# see <http://www.imagemagick.org/script/command-line-options.php#flatten>
convert image.png -flatten image.jpg

Converting all pixels of a given color to transparent

# we want to make white transparent
convert image.jpg -fuzz 5% -transparent white image.png

Some References

• Fred's ImageMagick Tidbits

• Convert PNG with transparency to JPG (Stackoverflow)

• [ImageMagick Snippets by Mohamed A. Hassan](https://gist.github.com/MohamedAlaa/d9b54cd856b3edce1510]

mozjpeg

compress JPEG images

Batch compress a bunch of JPG files in the current directory

for img in *.jpg; do mozjpeg -outfile /path/to/output/directory/$img $img; done

pngquant

compress PNG images

# this will compress image.png file and save it as a new file with name "image-fs8.png"
pngquant image.png

# change output filename suffix. This will result in a new file image.min.png
pngquant image.png --ext .min.png

# save converted files in different location, instead of current directory
pngquant image.png --output path/to/output/directory/image.png

# strip Image metadata
pngquant --strip image.png

# skip saving files if the size of compressed files are larger than original files
pngquant --skip-if-larger image.png

# specify min/max quality
# set image quality in range 0 (worst) to 100 (perfect)
# here we set the minimum image quality as 60 and maximum quality as 80
pngquant --quality=60-80 image.png

PDFtk

concatenate various PDF files

# this concatenates all pdf files in current directory, saving them as filename.pdf
pdftk *.pdf cat output filename.pdf

# concatenate specific PDF files
pdftk file1.pdf file2 cat output filename.pdf

# if you want to see what's happening,
# add the verbose option at the end of your command. This is true for any pdftk operation
pdftk *.pdf cat output filename.pdf verbose

rotate PDF

# rotate page 1 by 90 degrees clockwise
pdftk input.pdf cat 1east output output.pdf

# rotate page 1 by 90 degrees anti-clockwise
pdftk input.pdf cat 1west output output.pdf

# rotate page 1 by 180 degrees
pdftk input.pdf cat 1south output output.pdf

# rotate all pages 90 degrees clockwise
pdftk input.pdf cat 1-endeast output output.pdf

# rotate all pages 90 degrees anti-clockwise
pdftk input.pdf cat 1-endwest output output.pdf

# rotate all pages 180 degrees
pdftk input.pdf cat 1-endsouth output output.pdf

encrypt PDF

# No. 1: Encrypt a PDF using 128-bit strength (the default), withhold all permissions (the default)
pdftk file1.pdf output file1.128.pdf owner_pw your_password

# No. 2: Same as above, except printing is allowed
pdftk file1.pdf output file1.128.pdf owner_pw foo owner_pw your_password allow printing

No. 3: Same as No. 1, except password baz must also be used to open output PDF
pdftk file1.pdf output file1.128.pdf owner_pw foo user_pw baz

extract pages from PDF

# extract page 1, 4 and 5 from input.pdf, save them as one document, output.pdf
pdftk input.pdf cat 1 4 5 output output.pdf verbose

# same as above, but as separate files
for pages in {1,4,5};do pdftk input.pdf cat $pages output extracted-$pages.pdf verbose;done

# you can also extract a range, like page 3 to 15, or page 19 to the end of the document, or both:
pdftk input.pdf cat 3-15 output output.pdf verbose
pdftk input.pdf cat 19-end output output.pdf verbose
pdftk input.pdf cat 3-15 19-end output output.pdf verbose

# in one operation, you can extract pages from separate documents and combine them as one
# here, we have two files, file1.pdf and file2.pdf, and we're
# extracting pages 1-3 from file1.pdf, and extracting pages 4 to the end from file2.pdf
pdftk A=file1.pdf B=file2.pdf cat A1-3 B4-end output newfile.pdf verbose

"stamp" a PDF file with another PDF file

There are many practical use cases for this. For example, you want to add a watermark to a PDF, or you want to add a signature to a PDF.

# this usually works best if both files are of the same size
pdftk page_to_be_stamped.pdf stamp stamp.pdf output stamped.pdf

# same as above, but for a multi-page document
# see <https://www.pdflabs.com/docs/pdftk-man-page/#dest-op-multistamp>
pdftk document_to_be_stamped.pdf multistamp stamp.pdf output stamped.pdf

Ghostscript

compress pdf

Ghostscript is super powerful, and to be honest, the first time I encountered it I felt intimidated by the myriad of options available. Fortunately, Alfred Klomp has written an excellent wrapper around Ghostscript to reduce a PDF's file size. It comes as a BASH script called shrinkpdf, and has proved to be very handy for me over the years. I have even included it as part of my BASH setup script for Ubuntu servers

I normally add the script in a bin directory inside my $HOME directory, and ensure that

1. the script is executable (chmod +x ~/bin/shrinkpdf)

2. ~/bin/ is on my $PATH (export PATH="$PATH:$HOME/bin")

Then use it as follows:

shrinkpdf -o output.pdf input.pdf

# And an output resolution in DPI (default is 72 DPI) with the -r option
shrinkpdf -r 150 -o output.pdf input.pdf

OCRmyPDF

Add an OCR layer and convert to PDF/A

ocrmypdf input.pdf output.pdf

Add an OCR layer and output a standard PDF

ocrmypdf --output-type pdf input.pdf output.pdf

Correct page rotation

ocrmypdf --rotate-pages input.pdf output.pdf

Produce PDF and text file containing OCR text

ocrmypdf --sidecar output.txt input.pdf output.pdf

OCRmyPDF can also convert single images to PDFs on its own

# If the resolution (dots per inch, DPI) of an image is not set or is incorrect,
# it can be overridden with --image-dpi, e.g. --image-dpi 300
ocrmypdf image.png myfile.pdf

If you have multiple images, use img2pdf to convert the images to PDF. Here's an example, where you convert your images to PDFs, and then pipe the results to run ocrmypdf. The - tells OCRmyPDF to read standard input:

img2pdf my-images*.jpg | ocrmypdf - myfile.pdf

SVGO

Compress a single SVG file

# you can skip the optional -i argument
svgo -i input.svg -o output.svg

Compress multiple SVG files in a directory

svgo -f path/to/dir/with/svg/files -o path/to/dir/with/svg/output

It's worth mentioning that some of these commands are too verbose, and it's not easy to remember all the arguments and options for some of them. In such cases, I often find it useful to have BASH functions for some of these. For instance, I have one for encrypting PDFs:

# encrypt pdf, allow printing
encrypt_pdf() {
  encrypted_pdf="${1%.pdf}.128.pdf"
  pdftk "$1" output ${encrypted_pdf} owner_pw "$2" allow printing verbose

  # rename the files after encryption
  mv -v "$1" "${1%.pdf}_src.pdf"
  mv -v ${encrypted_pdf} "${encrypted_pdf%.128.pdf}.pdf"
}

I call it like this:

encrypt_pdf filename.pdf $(openssl rand -base64 12)

Hope you found this post useful! If you have any cool tricks and time-saving techniques, please share them in the comments!

My initial plan was to spend some time looking at different CS curricula from various high-profile universities – comparing the structure, looking at common themes and so on, but I realised that that would be quite a significant undertaking and I didn't have that much time – I needed to make a decision quickly and get started immediately. In the past, I have often wasted a lot of time over-analysing things and coming up with very intricate plans – only to later on miserably fail to implement them😒! Therefore, this time I decided not to waste time over-analysing and overthinking things, especially given that people smarter than I have already encountered this and done the work already! I mean, I already talked about teachyourselfcs.com in my previous post and I agree with their suggested 9 subjects as being essential for every practising software engineer. The OSSU curriculum is great, but I think it's too detailed and rigorous. It's very easy to lose focus and go down the rabbit hole. As I mentioned in my previous post, I have very limited free time, so I have to make the most of it. Following the OSSU curriculum would not be a prudent use of the limited time I have. 10 years ago, it would have made perfect sense to delve into the OSSU pathway, but things have changed now. A clearer, more succinct and optimum pathway is what I need, and teachyourselfcs.com does it for me.

While I'm set on following the teachyourselfcs.com approach as a guide, I will not necessarily follow the recommended videos/lectures suggested for each subject. I decided to tweak things a little bit and come up with my own structure based on what I was trying to achieve at the end of the day.

I would like to learn the foundational CS principles and concepts but I would also like to earn a qualification while doing so because God-willing, I'd like to enrol in a Computer Science Master's programme in the near future. The Folks at teachyourselfcs.com say that a Master's degree is not important. Well, I respect their view, but for me, I would like to pursue a Master's degree – not that I need it but I think it's something nice to have as there are certain kinds of clientele that specifically demand a Bachelor's or Master's degree.

In light of the foregoing, I will be selecting those courses that allow me to earn a certificate as that will greatly help me as I seek to gain admission to a Master's programme at some point in the future. This means my primary focus will be looking at the various available MOOCs such as Coursera, edX, Udacity and so on. So I went over to classcentral.com, created a profile and used it to find courses and make comparisons. Class Central is quite an excellent resource, you should check it out if you haven't already!

After making comparisons of the various platforms, I settled on Coursera because for $399 a year with the Coursera Plus subscription (I was fortunate to get a $100 Black Friday discount for the first year, so I actually paid $299 ☺️), I have a wide array of courses to choose from and the freedom to pursue them in my own time and at my own pace. I concluded that this was the most cost-effective way for me. The key thing is to manage my time wisely and pick the courses that would provide the most value.

Of course, Coursera isn't the solution for everything – I will certainly have to look at other platforms (and find the money 💵 to pay) for specific courses of interest.

I'm looking at a timeline of about 2 years to cover these 9 subjects. In the first year, I intend to focus on the first 4 items on the list:

1. Programming

2. Computer Architecture

3. Data Structures and Algorithms

4. Math for CS

However, before rushing to get started, and before I even paid for the Coursera Plus subscription, I decided to give myself a challenge by enrolling in a course with a free certificate (outside Coursera) and ensuring that I completed it within the indicative timeframe. I've enrolled in such courses (and programming tutorials) before – there are very very few that I've actually completed 🙈! I thought this would be a good indication of my readiness to do this. I told myself that if I struggled to complete this one course, then perhaps I wasn't quite ready to commit myself. I enrolled in the Starting Up course offered by Aalto University and MinnaLearn and I tweeted about this (it's been said that people tend to be more committed to their goals after they share them). I was able to complete it in just under 4 weeks, which was great because the indicative timeframe was 6 weeks. This gave me some confidence and helped me identify my weaknesses and plan my time, so I think I'm now ready to roll!

Because of the limited Black Friday deal (which I find out via ClassCentral), I actually ended up subscribing to Coursera Plus before I even completed the Starting Up course.

Since this is no trivial undertaking, involving learning lots of new stuff, I thought it necessary to spend some time learning how to learn. Therefore, before I delve into the 4 focus areas for the first year, I enrolled in Learning How to Learn: Powerful mental tools to help you master tough subjects, a highly recommended course. I know, I'm supposed to get started with CS already, right? Well, I need to make the most of the limited time available, which means I also need to master the technique of learning so I can learn new things effectively. So this is something important, and my future self will surely be most grateful that I did this!

I intend to begin the first-year journey with Fundamentals of Computing Specialization offered by Rice University. This covers 2 of the 4 subjects (Programming + Data Structures and Algorithms). There's an indicative completion timeframe of 6 months, we'll see how it goes – hopefully, I can do it in less time. I'll be reviewing progress and evaluating things along the way, let's see how it goes! Until the next time .... adios!

Photo by Braden Collum on Unsplash

I learnt how to code while still studying civil engineering, and continued to slowly (and inconsistently) learn one or two things after university, in my free time. Fast-forward to early 2022, a couple of months before my 37th birthday, I landed an opportunity to work as a software developer, and now, 8 months later, I'm at a point where I have to make an important decision on how to navigate this new career.

While it's possible to have a thriving career in tech without a CS degree, I think it's important to be well grounded in the fundamental CS concepts and principles. In the words of Oz Nova and Myles Byrne from teachyourselfcs.com

If you’re a self-taught engineer or bootcamp grad, you owe it to yourself to learn computer science.

And again,

There are 2 types of software engineer: those who understand computer science well enough to do challenging, innovative work, and those who just get by because they’re familiar with a few high level tools.

I don't want to be the type 2 guy 😀, so I have to do something about it!

What are my available options?

Physical or Online?

At this stage in my life, I do not have the luxury of going to university full time, so that's out of the question. This only leaves me with online options.

Pursue a degree, or learn on your own?

Now, do I want to pursue an actual CS degree, or do I just want to learn the core subjects on my own? Well, I think it's nice to have an actual CS degree. However, one has to

1. find a well accredited University with good rankings
2. have the money to pay for university education
3. have the time to actually invest into learning, doing assessments, exams, etc.

Item № 1 is not difficult. The challenge is № 2 and 3.

Most of the really good universities out there aren't cheap. There are some institutions which I've found to be affordable, but I'm not quite sure about credibility. I do not want to spend 3+ years pursuing an academic qualification which will only be recognized in very few places. That's a definite no-no for me!

If I'm going to enroll to study at a university, I have to dedicate about 25 hours per week towards studies, that's quite a lot of time for someone who has a family, has a full time job and other responsibilities.

So there has to be a balance between quality, affordability and time.

If degree – Bachelor's or Master's?

Having already earned a Bachelor's degree, the thought of going to pursue another Bachelor's degree kinda feels counter productive. It's more desirable to move forward and pursue a higher qualification.

There are a lot of excellent institutions that offer CS Master's degrees to people whose first degree is not necessarily a CS degree. However, I have also found that admission into these programs generally isn't easy. Further, I looked at a few curricula and saw that either

1. the content is generally "basic" and broad, not going in depth, since the target students are not coming from a CS background.
2. the content is advanced and rigorous, as one is expected to somehow already have the foundational concepts that are typically taught at undergraduate level.

Going for a program with a category 1 curriculum just for the sake of "having a Master's degree" would not do me any good. At this stage, it's hard for me to get into a university offering Category 2 curriculum, so I have to rule out a Master's degree for now. I need to be well grounded in the core CS concepts, so it would be better for me to pursue a Bachelor's degree.

However, even if I had the money, enrolling at a University to pursue an undergraduate CS degree may be difficult for me because of time constraints. I cannot commit to 25 hours per week at this point.

So, what do I do?

Only 1 option!

Well, clearly, my best option (unless you can convince me otherwise) is to self-study, in my own time, at my own pace. However, I have to do so in a well defined and structured manner, not haphazardly and only doing so when I feel like. What would really help is to be part of a community of other people in a similar situation, so that we study and learn together, encouraging and challenging each other along the way.

Fortunately, there are many resources and guidelines out there to help people like me pursue a CS education on their own. I have already referred to teachyourselfcs.com, which suggests that one studies the following nine core subjects, aiming for 100-200 hours of study of each, then revisiting favorites throughout one's career:

1. Programming
2. Computer Architecture
3. Algorithms and Data Structures
4. Math for CS
5. Operating Systems
6. Computer Networking
7. Databases
8. Languages and Compilers
9. Distributed Systems

There's also the OSSU curriculum, which is described as a complete education in computer science using online materials. OSSU has a discord server where one can interact with other OSSU students.

Looks like I'm sorted, right? Well, not quite!

Next steps

I need to look at the various guidelines and curricula, also comparing with actual CS programmes at selected universities.

I need to think critically about the end goal and structure my learning accordingly.

I'll put up a follow-up post where I document what I'll come up with — essentially, I'll be talking about building my own CS degree!

Until then, adios! 👋

Cover image by Dom Fou on Unsplash

import os
from datetime import datetime
from django.core.files.base import File
from wagtail.documents.models import Document

def create_wagtail_document(f, title):
    """
    programmatically create a wagtail document which will
    automatically be available in the Documents section of wagtailadmin
    """
    doc_file = File(open(f, "rb"), name=os.path.basename(f))
    doc = Document(
        title=title,
        file=doc_file,
    )
    doc.save()

f = os.path.join("your_file")
today = datetime.now().strftime("%Y-%b-%d-%a")
now = datetime.now().strftime("%-I:%M%p")
doc_title = f"My awesome document created on {today} at {now}"
create_wagtail_document(f, doc_title)

The solution was inspired by Pieter Claerhout's approach in Programatically importing images in Wagtail.

In this post, I present a list of VS Code extensions that I generally depend on, and always ensure that they are installed in my favourite text editor! So, without further ado, here we go ...

General Purpose ⚙️

Spell Checking ✍🏽

• streetsidesoftware.code-spell-checker – Code Spell Checker › Spelling checker for source code

Comments 💬

• aaron-bond.better-comments – Better Comments › Improve your code commenting by annotating with alert, informational, TODOs, and more!

Changelogs 📜

• axetroy.vscode-changelog-generator – changelog-generator › An extension to generate changelog.

Add a splash of colour 🎨

• CoenraadS.bracket-pair-colorizer-2 – Bracket Pair Colorizer 2 › A customizable extension for colorizing matching brackets
• kamikillerto.vscode-colorize – colorize › A vscode extension to help visualize css colors in files.
• naumovs.color-highlight – Color Highlight › Highlight web colors in your editor
• oderwat.indent-rainbow – indent-rainbow › Makes indentation easier to read

Lorem ipsum ❇️

• deerawan.vscode-faker – vscode-faker › Generate fake data for name, address, lorem ipsum, commerce and much more

Coding style 📄

• EditorConfig.EditorConfig – EditorConfig for VS Code › EditorConfig Support for Visual Studio Code
• esbenp.prettier-vscode – Prettier - Code formatter › Code formatter using prettier
• HookyQR.beautify – Beautify › Beautify code in place for VS Code

Code execution 🚀

• formulahendry.code-runner – Code Runner › Run C, C++, Java, JS, PHP, Python, Perl, Ruby, Go, Lua, Groovy, PowerShell, CMD, BASH, F#, C#, VBScript, TypeScript, CoffeeScript, Scala, Swift, Julia, Crystal, OCaml, R, AppleScript, Elixir, VB.NET, Clojure, Haxe, Obj-C, Rust, Racket, Scheme, AutoHotkey, AutoIt, Kotlin, Dart, Pascal, Haskell, Nim, ...

Metrics 📈

• WakaTime.vscode-wakatime – WakaTime › Metrics, insights, and time tracking automatically generated from your programming activity.

Git 🗃️

• rubbersheep.gi – gi › Generating .gitignore files made easy

Screen capture 📸

• pnp.polacode – Polacode › 📸 Polaroid for your code

Miscellaneous Tools 🛠️

• alefragnani.Bookmarks – Bookmarks › Mark lines and jump to them
• Gruntfuggly.todo-tree – Todo Tree › Show TODO, FIXME, etc. comment tags in a tree view
• humao.rest-client – REST Client › REST Client for Visual Studio Code
• kisstkondoros.vscode-gutter-preview – Image preview › Shows image preview in the gutter and on hover
• tombonnike.vscode-status-bar-format-toggle – Formatting Toggle › A VS Code extension that allows you to toggle the formatter (Prettier, Beautify, …) ON and OFF with a simple click.
• wmaurer.vscode-jumpy – jumpy › Jumpy provides fast cursor movement, inspired by Atom's package of the same name.

Themes 💄

• akamud.vscode-theme-onedark – Atom One Dark Theme › One Dark Theme based on Atom
• PKief.material-icon-theme – Material Icon Theme › Material Design Icons for Visual Studio Code
• sdras.night-owl – Night Owl › A VS Code theme for the night owls out there. Now introducing Light Owl theme for daytime usage. Decisions were based on meaningful contrast for reading comprehension and for optimal razzle dazzle. ✨
• vscode-icons-team.vscode-icons – vscode-icons › Icons for Visual Studio Code

Language Support ✨

• asciidoctor.asciidoctor-vscode – AsciiDoc › Provides rich language support for AsciiDoc.
• attilabuti.vscode-mjml – MJML › MJML preview, lint, compile for Visual Studio Code.
• bibhasdn.django-html – Django Template › Django template language support for Visual Studio Code
• bungcip.better-toml – Better TOML › Better TOML Language support
• Dart-Code.dart-code – Dart › Dart language support and debugger for Visual Studio Code.
• Dart-Code.flutter – Flutter › Flutter support and debugger for Visual Studio Code.
• DotJoshJohnson.xml – XML Tools › XML Formatting, XQuery, and XPath Tools for Visual Studio Code
• GrapeCity.gc-excelviewer – Excel Viewer › View Excel spreadsheets and CSV files within Visual Studio Code workspaces.
• IBM.output-colorizer – Output Colorizer › Syntax highlighting for log files
• idleberg.nsis – NSIS › Language syntax, IntelliSense and build system for Nullsoft Scriptable Install System (NSIS)
• lextudio.restructuredtext – reStructuredText › reStructuredText language support (RST/ReST linter, preview, IntelliSense and more)
• mikestead.dotenv – DotENV › Support for dotenv file syntax
• ms-dotnettools.csharp – C# › C# for Visual Studio Code (powered by OmniSharp).
• ms-vscode.cmake-tools – CMake Tools › Extended CMake support in Visual Studio Code
• ms-vscode.cpptools – C/C++ › C/C++ IntelliSense, debugging, and code browsing.
• naco-siren.gradle-language – Gradle Language Support › Add Gradle language support for Visual Studio Code
• NativeScript.nativescript – NativeScript › NativeScript support for Visual Studio Code
• redhat.vscode-yaml – YAML › YAML Language Support by Red Hat, with built-in Kubernetes syntax support
• redhat.vscode-xml – XML › XML Language Support by Red Hat
• samuelcolvin.jinjahtml – Better Jinja › Syntax highlighting for jinja(2) including HTML, Markdown, YAML, Ruby and LaTeX templates
• shanoor.vscode-nginx – nginx.conf › Syntax highlighter for nginx conf files.
• Syler.sass-indented – Sass › Indented Sass syntax Highlighting, Autocomplete & Formatter
• twxs.cmake – CMake › CMake langage support for Visual Studio Code
• william-voyek.vscode-nginx – NGINX Configuration › Syntax highlighting for NGINX configuration files
• xshrim.txt-syntax – Txt Syntax › highlight text files(.txt, .out .tmp, .log, .ini, .cnf ...) and provide general utility tools for text documents
• XadillaX.viml – VimL (Vim Language, Vim Script) › Vim Script language support for VSCode.

Snippets 🎉

• bibhasdn.django-snippets – Django Snippets › Common Django snippets for everyday use
• Keno.uikit-3-snippets – UIkit 3.0 Snippets › UIkit 3.0 Snippets based on official documentation
• msaelices.nativescript-vue-snippets – NativeScript-Vue Snippets › Snippets for Telerik's NativeScript mobile framework with Vue
• sdras.vue-vscode-snippets – Vue VSCode Snippets › Snippets that will supercharge your Vue workflow
• tsvetan-ganev.nativescript-xml-snippets – NativeScript XML Snippets › XML snippets for Telerik's NativeScript cross-platform mobile applications development framework.
• xabikos.JavaScriptSnippets – JavaScript (ES6) code snippets › Code snippets for JavaScript in ES6 syntax

Containers 🚢

• ms-azuretools.vscode-docker – Docker › Makes it easy to create, manage, and debug containerized applications.
• ms-vscode-remote.remote-containers – Remote - Containers › Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set.

Python 🐍

• ms-python.python – Python › IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more.
• ms-python.vscode-pylance – Pylance › A performant, feature-rich language server for Python in VS Code
• ms-toolsai.jupyter – Jupyter › Jupyter notebook support, interactive programming and computing that supports Intellisense, debugging and more.

TeX ✒️

• geoffkaile.latex-count – latex-count › A word counter for latex files (.tex)
• James-Yu.latex-workshop – LaTeX Workshop › Boost LaTeX typesetting efficiency with preview, compile, autocomplete, colorize, and more.

Web development 🌐

• christian-kohler.npm-intellisense – npm Intellisense › Visual Studio Code plugin that autocompletes npm modules in import statements
• ecmel.vscode-html-css – HTML CSS Support › CSS Intellisense for HTML
• dbaeumer.vscode-eslint – ESLint › Integrates ESLint JavaScript into VS Code.
• eg2.vscode-npm-script – npm › npm support for VS Code
• lonefy.vscode-JS-CSS-HTML-formatter – JS-CSS-HTML Formatter › Format, prettify and beautify JS, CSS, HTML code by using shortcuts, context menu or CLI
• ms-vscode.sublime-keybindings – Sublime Text Keymap and Settings Importer › Import Sublime Text settings and keybindings into VS Code.
• ms-vscode.vscode-typescript-tslint-plugin – TSLint › TSLint support for Visual Studio Code
• ritwickdey.LiveServer – Live Server › Launch a development local Server with live reload feature for static & dynamic pages
• sidthesloth.html5-boilerplate – HTML Boilerplate › A basic HTML5 boilerplate snippet generator.
• stylelint.vscode-stylelint – stylelint › Modern CSS/SCSS/Less linter
• webhint.vscode-webhint – webhint › Run webhint in Visual Studio Code.
• Zignd.html-css-class-completion – IntelliSense for CSS class names in HTML › CSS class name completion for the HTML class attribute based on the definitions found in your workspace.

Markdown 📝

• DavidAnson.vscode-markdownlint – markdownlint › Markdown linting and style checking for Visual Studio Code
• ms-vscode.Theme-MarkdownKit – Markdown Theme Kit › Theme Kit for VS Code optimized for Markdown. Based on the TextMate themes.
• ms-vscode.wordcount – Word Count › Markdown Word Count Example - a status bar contribution that reports out the number of works in a Markdown document as you interact with it.
• yzhang.markdown-all-in-one – Markdown All in One › All you need to write Markdown (keyboard shortcuts, table of contents, auto preview and more)

Vue.js 💚

• sdras.vue-vscode-extensionpack – Vue VS Code Extension Pack › A collection of extensions for working with Vue Applications in VS Code
• octref.vetur – Vetur › Vue tooling for VS Code

Android 📱

• jeroen-meijer.pubspec-assist – Pubspec Assist › Easily add and update dependencies to your Dart and Flutter project.
• richardwillis.vscode-gradle – Gradle Tasks › Run Gradle tasks in VS Code

Well, there you have it! What VS Code extensions do you depend on that aren't on this list? Do you have alternatives to some of the extensions I've listed? Well, let me know in the comments below 🙂.

Contents

• The problem
• The solution
  • 0. First things first
  • 1. Update the DailyReflectionPage model
  • 2. Create a new Task
  • 3. Email Notifications
• Demo

The problem

Suppose we have an organization that annually publishes a booklet consisting of daily reflections. This booklet is published towards the end of the year, for use in the coming year. The daily reflections are written by different authors, and we would like to have a publishing workflow where, for instance, the daily reflections from January to June are reviewed by one group, and those from July to December are reviewed by another group, as illustrated below:

We have a DailyReflectionPage Model with a reflection_date field that forms the basis for the Page's slug, which is in the form YYYY-MM-DD. Here's an extract of the Page model:

class DailyReflectionPage(Page):
    """
    The Daily Reflection Model
    """
    ...
    ...

    reflection_date = models.DateField("Reflection Date", max_length=254)
    ...
    ...
    @cached_property
    def date(self):
        """
        Returns the Reflection's date as a string in %Y-%m-%d format
        """
        fmt = "%Y-%m-%d"
        date_as_string = (self.reflection_date).strftime(fmt)
        return date_as_string      
    ...
    ...
    def full_clean(self, *args, **kwargs):
        # first call the built-in cleanups (including default slug generation)
        super(DailyReflectionPage, self).full_clean(*args, **kwargs)

        # now make your additional modifications
        if self.slug is not self.date:
            self.slug = self.date
    ...
    ...

If you want to see the complete Page Model, then have a look at the models.py file in the reflections app. This link points to a commit before implementation of the solution outlined herein.

The solution

I initially created a gist showing the full implementation of LB's solution. It only consists of themodels.py file, so if you just want to jump straight to the basic solution, then you can look at it. However, if you would like to see a more complete solution within the wider context of an actual Wagtail project, you can have a look at the sample Wagtail project I created to accompany this post. The source code is available at github.com/engineervix/wagtail-branching-workflows.

0. First things first

If you would like to follow along, starting from the point before implementing the solution, you can clone the repo and checkout commit 5745df

git clone https://github.com/engineervix/wagtail-branching-workflows.git
cd wagtail-branching-workflows
git checkout 5745df

The project uses docker and docker-compose, so make sure that these are installed and working on your machine:

# check that you have docker on your machine
docker -v

# check that you have docker-compose on your machine
docker-compose -v

Then create the required .env files:

cp -v app/.envs/.dev.env.sample app/.envs/.dev.env
cp -v app/.envs/.test.env.sample app/.envs/.test.env

Build the images and spin up the containers:

docker-compose up -d --build

You'll have to wait a few seconds for some processes to initialize / run (postgres, database migrations, browser-sync, Django server, etc.). You can check the status via

docker-compose logs web

When all set, you should see something like this:

web_1  | Performing system checks...
web_1  | 
web_1  | [Browsersync] Proxying: http://127.0.0.1:8000
web_1  | [Browsersync] Access URLs:
web_1  |  -----------------------------------
web_1  |        Local: http://localhost:3000
web_1  |     External: http://172.19.0.3:3000
web_1  |  -----------------------------------
web_1  |           UI: http://localhost:3001
web_1  |  UI External: http://localhost:3001
web_1  |  -----------------------------------
web_1  | [Browsersync] Watching files...
web_1  | System check identified no issues (0 silenced).
web_1  | 
web_1  | Django version 3.2.8, using settings 'config.settings.dev'
web_1  | Development server is running at http://0.0.0.0:8000/
web_1  | Using the Werkzeug debugger (http://werkzeug.pocoo.org/)
web_1  | Quit the server with CONTROL-C.
web_1  | [Browsersync] Reloading Browsers... (buffered 2 events)
web_1  |  * Debugger is active!
web_1  |  * Debugger PIN: 104-102-219

You can now proceed to create a superuser:

docker-compose exec web ./manage.py createsuperuser

Load initial data:

docker-compose exec web ./manage.py load_initial_data

This initial data includes 6 users with the following details:

You can access the dev server at http://127.0.0.1:3009. This project uses MailDev for viewing and testing emails generated during development. The MailDev server is accessible at http://localhost:1089.

1. Update the DailyReflectionPage model

First, we add a method called date_in_first_semester:

@cached_property
def date_in_first_semester(self):
    """
    Returns True if Reflection's date
    is in the first half of the year
    """
    month = (self.reflection_date).month
    return month <= 6

Then, we add a method called get_approval_group_key which will return a simple Boolean or maybe something like 'A' or 'B' (feel free to suit this to your liking).

def get_approval_group_key(self):
    # custom logic here that checks all the date stuff
    if self.date_in_first_semester:
        return "A"
    return "B"

2. Create a new Task

We will now create a new Task that extends the Wagtail Task class.

The following official Wagtail resources are essential in gaining an understanding of what's going on:

1. how to add a new Task Type
2. Task model reference.
3. source code of the built-in GroupApprovalTask

# only **additional** imports are shown here
from django import forms
...
...
# from wagtail.core.models import Page
from wagtail.core.models import Group, Page, Task, TaskState, WorkflowState
...
...

class SplitGroupApprovalTask(Task):

    ## note: this is the simplest approach, two fields of linked groups, you could further refine this approach as needed.

    groups_a = models.ManyToManyField(
        Group,
        verbose_name="for Jan - June Daily Reflections",
        help_text="Pages at this step in a workflow will be moderated or approved by these groups of users",
        related_name="split_task_group_a",
    )
    groups_b = models.ManyToManyField(
        Group,
        verbose_name="for Jul - Dec Daily Reflections",
        help_text="Pages at this step in a workflow will be moderated or approved by these groups of users",
        related_name="split_task_group_b",
    )

    admin_form_fields = Task.admin_form_fields + ["groups_a", "groups_b"]
    admin_form_widgets = {
        "groups_a": forms.CheckboxSelectMultiple,
        "groups_b": forms.CheckboxSelectMultiple,
    }

    def get_approval_groups(self, page):
        """This method gets used by all checks when determining what group to allow/assign this Task to"""

        # recommend some checks here, what if `get_approval_group` is not on the Page?

        # here's a simple check
        if hasattr(page.specific, "get_approval_group_key"):
            approval_group = page.specific.get_approval_group_key()
        else:
            # arbitrarily assign to group A 
            # (you could instead do something else)
            approval_group = "A"

        if approval_group == "A":
            return self.groups_a

        return self.groups_b

    # each of the following methods will need to be implemented, all checking for the correct groups for the Page when called
    # def start(self, ...etc)
    # def user_can_access_editor(self, ...etc)
    # def page_locked_for_user(self, ...etc)
    # def user_can_lock(self, ...etc)
    # def user_can_unlock(self, ...etc)
    # def get_task_states_user_can_moderate(self, ...etc)

    def start(self, workflow_state, user=None):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        approval_groups = self.get_approval_groups(workflow_state.page)

        if workflow_state.page.locked_by:
            # If the person who locked the page isn't in one of the groups, unlock the page
            # if not workflow_state.page.locked_by.groups.filter(id__in=self.groups.all()).exists():
            if not approval_groups.filter(id__in=self.groups.all()).exists():
                workflow_state.page.locked = False
                workflow_state.page.locked_by = None
                workflow_state.page.locked_at = None
                workflow_state.page.save(
                    update_fields=["locked", "locked_by", "locked_at"]
                )

        return super().start(workflow_state, user=user)

    def user_can_access_editor(self, page, user):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        approval_groups = self.get_approval_groups(page)

        # return self.groups.filter(id__in=user.groups.all()).exists() or user.is_superuser
        return (
            approval_groups.filter(id__in=user.groups.all()).exists()
            or user.is_superuser
        )

    def page_locked_for_user(self, page, user):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        approval_groups = self.get_approval_groups(page)

        # return not (self.groups.filter(id__in=user.groups.all()).exists() or user.is_superuser)
        return not (
            approval_groups.filter(id__in=user.groups.all()).exists()
            or user.is_superuser
        )

    def user_can_lock(self, page, user):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        approval_groups = self.get_approval_groups(page)

        # return self.groups.filter(id__in=user.groups.all()).exists()
        return approval_groups.filter(id__in=user.groups.all()).exists()

    def user_can_unlock(self, page, user):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        # approval_groups = self.get_approval_groups(page)
        return False

    def get_actions(self, page, user):
        # essentially a copy of this method on `GroupApprovalTask` but with the ability to have a dynamic 'group' returned.
        approval_groups = self.get_approval_groups(page)

        if (
            approval_groups.filter(id__in=user.groups.all()).exists()
            or user.is_superuser
        ):
            return [
                ("reject", "Request changes", True),
                ("approve", "Approve", False),
                ("approve", "Approve with comment", True),
            ]

        return super().get_actions(page, user)

    def get_task_states_user_can_moderate(self, user, **kwargs):

        # not a very DRY approach, but it works!

        if user.is_superuser:
            return TaskState.objects.filter(
                status=TaskState.STATUS_IN_PROGRESS, task=self.task_ptr
            )
        elif self.groups_a.filter(id__in=user.groups.all()).exists():
            return TaskState.objects.filter(
                status=TaskState.STATUS_IN_PROGRESS,
                task=self.task_ptr,
                workflow_state__in=WorkflowState.objects.filter(
                    page__in=DailyReflectionPage.objects.filter(
                        reflection_date__month__lte=6
                    )
                ),
            )
        elif self.groups_b.filter(id__in=user.groups.all()).exists():
            return TaskState.objects.filter(
                status=TaskState.STATUS_IN_PROGRESS,
                task=self.task_ptr,
                workflow_state__in=WorkflowState.objects.filter(
                    page__in=DailyReflectionPage.objects.filter(reflection_date__month__gt=6)
                ),
            )
        else:
            return TaskState.objects.none()

    @classmethod
    def get_description(cls):
        return "Groups are assigned to approve this task based on reflection month"

    class Meta:
        verbose_name = "reflection-month-dependent approval task"
        verbose_name_plural = "reflection-month-dependent approval tasks"

3. Email Notifications

By default, email notifications are sent upon workflow submission, approval and rejection, and upon submission to the built-in GroupApprovalTask. We will replicate this behaviour in our SplitGroupApprovalTask. This is a three-step process:

1. Create a mail.py file within our reflections app and add the following classes:
   • A base Notifier to send updates for SplitGroupApprovalTask events
   • A notifier to send updates for SplitGroupApprovalTask submission events
2. Create a signal_handlers.pyfile within our reflections app. In here, we instantiate the notifier, and connect it to the task_submitted signal via the register_signal_handlers() method.
3. Run register_signal_handlers() upon loading our reflections app

Here's the code:

# Step 1: reflections/mail.py

from django.conf import settings
from django.contrib.auth import get_user_model
from wagtail.admin.mail import EmailNotificationMixin, Notifier
from wagtail.core.models import TaskState

from mysite.reflections.models import SplitGroupApprovalTask


class BaseSplitGroupApprovalTaskStateEmailNotifier(EmailNotificationMixin, Notifier):
    """A base notifier to send updates for SplitGroupApprovalTask events"""

    def __init__(self):
        # Allow TaskState to send notifications
        super().__init__((TaskState))

    def can_handle(self, instance, **kwargs):
        if super().can_handle(instance, **kwargs) and isinstance(
            instance.task.specific, SplitGroupApprovalTask
        ):
            # Don't send notifications if a Task has been cancelled and then resumed - ie page was updated to a new revision
            return not TaskState.objects.filter(
                workflow_state=instance.workflow_state,
                task=instance.task,
                status=TaskState.STATUS_CANCELLED,
            ).exists()
        return False

    def get_context(self, task_state, **kwargs):
        context = super().get_context(task_state, **kwargs)
        context["page"] = task_state.workflow_state.page
        context["task"] = task_state.task.specific
        return context

    def get_recipient_users(self, task_state, **kwargs):

        triggering_user = kwargs.get("user", None)

        approval_groups = task_state.task.specific.get_approval_groups(
            task_state.workflow_state.page
        )

        group_members = get_user_model().objects.filter(
            groups__in=approval_groups.all()
        )

        recipients = group_members

        include_superusers = getattr(
            settings, "WAGTAILADMIN_NOTIFICATION_INCLUDE_SUPERUSERS", True
        )
        if include_superusers:
            superusers = get_user_model().objects.filter(is_superuser=True)
            recipients = recipients | superusers

        if triggering_user:
            recipients = recipients.exclude(pk=triggering_user.pk)

        return recipients


class SplitGroupApprovalTaskStateSubmissionEmailNotifier(
    BaseSplitGroupApprovalTaskStateEmailNotifier
):
    """A notifier to send updates for SplitGroupApprovalTask submission events"""

    notification = "submitted"

# Step 2: reflections/signal_handlers.py

from wagtail.core.signals import task_submitted

from mysite.reflections.mail import (
    SplitGroupApprovalTaskStateSubmissionEmailNotifier,
)

task_submission_email_notifier = SplitGroupApprovalTaskStateSubmissionEmailNotifier()


def register_signal_handlers():
    task_submitted.connect(
        task_submission_email_notifier,
        dispatch_uid="task_submitted_email_notification",
    )

# Step 3: reflections/apps.py

from django.apps import AppConfig


class ReflectionsConfig(AppConfig):
    name = "mysite.reflections"

    def ready(self):
        from .signal_handlers import register_signal_handlers

        register_signal_handlers()

You can check out the Extending Wagtail --> How to add new Task types --> Adding notifications section of the Wagtail Docs for more information.

In my initial stages of trying to figure this out, the example source code in the docs was not up to date. I submitted a PR to fix this, and it was merged.

You might also wanna have a look at the workflow settings reference for further workflow customization.

Well, that's about it! You can run migrations, log into the Wagtail Admin, create some content, some workflows and tasks ... check out the video below to see this in action!

Demo

I see myself as one of those people who are better able to express themselves in writing rather than speaking. Growing up as an introvert, I discovered that I enjoyed expressing my thoughts and ideas on paper. In my High School days, I would usually top my class in English composition. I remember once getting a score of 18/20 in an essay, while most people scored in the 10 – 13 range!

My tech stack choices are sometimes heavily influenced by the quality of the documentation. I actually enjoy writing docs myself, and I tend to put some decent effort into the process. I think this is well reflected in some of my projects, for example, readme-coverage-badger, cookiecuter-wagtail-vix and ubuntu-server-setup.

Why write?

When I got into software development, I realized that I learnt a lot from reading, and a huge part of that reading was from other developers' blogs! Sometimes, you read a tutorial, and as you follow along, you discover that you have to make so many modifications to get things working on your machine. Or perhaps you are trying to solve a specific dev problem which requires some bit of research – scouring the inter-webs for information and combining this and that to build a solution to your problem. What happens when you have to do the same thing again, say, two years from now? "Well, go back to your old code" – you would say! But we all know that future you may look at the code (s)he wrote in the past and have absolutely no clue what is going on!

This is why I thought that it would be a good idea to start a blog, so that I could have a place where I could document the things I learned, not only for the sake of future me, but also for others who might encounter the same challenges I faced.

Early attempts at technical blogging

I setup my first blog sometime in 2012/2013, using Wordpress. I wrote a few articles (less than 10!), but was very inconsistent, and I ended up shutting down the site.

After learning Python, I decided to build my own blog, and settled on Pelican – a powerful and popular python-based static site generator. This was around 2014/2015. Again, I wrote very few articles, mostly haphazardly and without a proper plan and schedule.

I started getting frustrated with my failure to write consistently and maintain a decent level of organization around my writing and learning. After a long period of inactivity on the blog, I embarked on a challenge to rebrand myself and rebuild my blog. I even wanted to write my own Pelican theme and make it open source. I got started developing a custom theme using UIkit, but the project went into development hell and I ended up abandoning the whole notion of building my own blog!

I was getting super busy at work and I didn't have sufficient time to devote to building my own blog. I learnt about platforms like medium.com and dev.to, where I was quick to sign up, but never wrote a single article on either platform! Notwithstanding, I still had the desire to write and have my own space on the web where I could pen down not only my thoughts and ideas, but also share what I learnt in my dev journey.

Hello Hashnode

Well, fast-forward to the year 2020, and I discover Hashnode through @sikaili99! I was immediately blown away – custom domain, a fantastic and growing dev community, SEO, I get to own my content, GitHub integration ... Wow! Such awesome! I thought to myself, "this is it, it's time to get serious!"

Towards the end of 2020, I sat down and did some bit of planning to decide how I was gonna proceed, seeing that my previous writing efforts were mostly unsuccessful. Here are two important things I did:

• I thought carefully about the content & focus of my technical blog. Because I love Python, I decided that I would largely focus on Python and related technologies. This is why I chose importthis.tech as my domain!
• I resolved to start by writing at least 12 articles in 2021 – that's one article per month.

This is my 8^th article, and I wrote it in the 9^th month of 2021! Not too bad, right?

Motivation

I have received some positive feedback on a few of my articles, and this has inspired me to keep writing. Getting to sit down to write an article can be a challenge, but the feeling of satisfaction that I get when I hit that Publish button is exhilarating, to say the least! It's like running or any other physical exercise – getting up to go for a run or hit the gym isn't easy, but we all feel good when we're back from exercise, and are often glad that we went!

Being a self-taught developer seeking to switch careers (from civil engineering to software engineering), I see technical writing as a way to demonstrate my abilities and document my growth in the craft. I see this as one of the means to showcase my work to a potential employer!

Further, the prospect of actually earning income from technical writing is another motivation factor. I have read a lot of stories of developers like myself who earn passive income from their writing.

Shout-outs

This is my first year of maintaining a decent level of consistency in my technical writing, and I am learning a lot of things along the way. All of this would not have been possible without Hashnode. You guys are the best, and I'm really grateful for the many opportunities that you've brought my way, including the Technical Writing Bootcamp, which I attended for the first time on September 13^th, 2021. I was really encouraged not only by @didicodes and @tanoaksam, but also by my fellow participants, most notably @winnekes – who's story is very inspiring!

Where do we go from here?

As the year comes to a close, I'll need to sit down again to reflect on how I fared this year, address some of my weaknesses and work on building my personal brand. In the meantime, the writing continues ... 🚀

Cover image by Cathryn Lavery on Unsplash

In this post, I present a list of Node.js tools that I generally depend on, and always ensure that they are installed globally on my development machine. So, without further ado, here we go ...

Web development 🌐

• @vue/cli – Standard Tooling for Vue.js Development
• @vue/cli-init – vue init command addon for @vue/cli. This is an alias to the old vue-cli@2.x.
• browser-sync – Keep multiple browsers & devices in sync when building websites.
• caniuse-cmd – a caniuse.com command line tool
• concurrently – Run multiple commands concurrently. I typically use this to simultaneously run the Django dev server, gulp and maildev.
• clean-css-cli – a command-line interface to clean-css - fast and efficient CSS optimizer for Node.js.
• express-generator – Express application generator tool,
• firebase-tools – Firebase CLI Tools for testing, managing, and deploying your Firebase project from the command line
• grunt-cli – JavaScript Task Runner. I have a couple of old projects that use Grunt, but I have since transitioned to Gulp, which seems to be more actively maintained and has wider plugin support.
• gulp-cli – A toolkit to automate & enhance your workflow.
• html-minifier – a highly configurable, well-tested, JavaScript-based HTML minifier.
• lerna – A tool for managing JavaScript projects with multiple packages.
• lite-server – Lightweight development only node server that serves a web app, opens it in the browser, refreshes when html or Javascript change, injects CSS changes using sockets, and has a fallback page when a route is not found
• local-cors-proxy – Simple proxy to bypass CORS issues. This was built as a local dev only solution to enable prototyping against existing APIs without having to worry about CORS.
• maildev – SMTP Server + Web Interface for viewing and testing emails during development.
• nodemon – a tool that helps develop Node.js based applications by automatically restarting the node application when file changes in the directory are detected.
• parcel-bundler – Blazing fast, zero configuration web application bundler
• pm2 – a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.
• prettier – an opinionated code formatter
• sass – CSS with superpowers
• serve – Static file serving and directory listing
• typescript – JavaScript with syntax for types.
• uglify-js – a JavaScript parser, minifier, compressor and beautifier toolkit.

Mobile app development 📱

• @ionic/cli – command-line interface for developing Ionic apps.
• @quasar/cli – I've placed Quasar under the mobile app development category because that's what I've used it for. However, it is a multi-purpose tool. In fact, Quasar’s motto is: “write code once and simultaneously deploy it as a website, a Mobile App and/or an Electron App”.
• @quasar/icongenie – outputs a set of SQUARE favicons, webicons, pwa-icons and electron-icons as well as iOS, Windows Store and MacOS icons from an original 1240x1240 square icon that retains transparency and also minifies the assets. It will also create splash screens for Cordova/Capacitor and even a minified svg.
• cordova – an open-source mobile development framework which allows you to use standard web technologies (HTML5, CSS3, and JavaScript) for cross-platform development.
• nativescript – access native APIs from JavaScript directly. The framework provides iOS and Android runtimes for rich mobile development.

Git repository management 💻

• commitizen – Simple commit conventions for internet citizens
• conventional-changelog-cli – Generate a changelog from git metadata
• dependabot-config-generator – CLI tool for Dependabot config generate
• standard-version – A utility for versioning using semver and CHANGELOG generation powered by Conventional Commits.

Screen recording / terminal capture 📹

• asciicast2gif – a tool for generating GIF animations from asciicast files recorded by asciinema.
• gifify – Convert any video file to an optimized animated GIF. This tool is no longer maintained, so I've switched to gifski, a Rust-based GIF encoder based on libimagequant
• svg-term-cli – Share terminal sessions as razor-sharp animated SVG everywhere.
• terminalizer – Record your terminal and generate animated gif images or share a web player.

Document processing / conversion 🗎

• doctoc – Generates table of contents for markdown files inside local git repository.
• mdpdf – A command line markdown to pdf converter with support for page headers, footers, and custom stylesheets. Mdpdf is incredibly configurable and has a JavaScript API for more extravogant usage.
• puppeteer-pdf – HTML to PDF from the command line with Puppeteer

CV / Resume generation 📃

• fluentcv – a dev-friendly, local-only Swiss Army knife for resumes and CVs. It is the corporate-friendly fork of HackMyResume.
• hackmyresume – Create polished résumés and CVs in multiple formats from your command line or shell. Author in clean Markdown and JSON, export to Word, HTML, PDF, LaTeX, plain text, and other arbitrary formats

Image compression 🖻

• imagemin-cli – Minify images seamlessly
• imagemin-advpng – AdvPNG plugin for imagemin
• imagemin-jpegtran – jpegtran plugin for imagemin
• imagemin-mozjpeg – Imagemin plugin for mozjpeg
• imagemin-optipng – optipng plugin for imagemin
• imagemin-pngcrush – pngcrush plugin for imagemin
• imagemin-pngquant – Imagemin plugin for pngquant
• mozjpeg – a production-quality JPEG encoder that improves compression while maintaining compatibility with the vast majority of deployed decoders
• svgo – Node.js tool for optimizing SVG files. I often also use a similar Python tool called Scour. I'd run both tools on one file and get the smaller resulting file!

Well, there you have it! What Node.js tools do you regularly use that aren't on this list? Do you have alternatives to some of the tools I've listed? Well, let me know in the comments below 🙂.

Many developers use services like coveralls.io or codecov.io for test coverage analysis and reporting. These services are free for open source projects, but require a monthly subscription for private repos. Many times, we work on private repos, and we wanna be able to automatically have coverage badges in our READMEs. What if you are unable to pay such subscription fees, or maybe you don't want to use a SaaS? Your solution becomes to generate your own badge!

Now, there are so many excellent coverage badge generation tools out there, but I couldn't find one to suit my needs. All the existing python tools (for example, coverage-badge) I had come across ended at generating SVG/PNG files/strings/Base64 images. What you do with this remains entirely up to you. Having used istanbul-badges-readme on Javascript projects, I wanted a python alternative but couldn't find anything, so I decided to write one myself!

Initially, I wrote a python script for use on a specific project. This script would be placed in a scripts or utils folder in the project root, and, I would use npm scripts to make it easy to run the python script, like this

npm readme-cov

I thought to myself, "what happens when I'm working on another project? Am I going to have to be copying this script around from project to project? No way!" And so I decided to package the script and publish it to PyPI so that I (and others who might find it useful) can just pip install it and have it in your PATH within a couple seconds, ready for use.

The process

Packaging 101

At this point, I knew very little about python packaging, so I had to do a bit of digging to learn how it works and how I could package and distribute my project. To start with, I used audreyfeldroy/cookiecutter-pypackage as the base for my package. It comes with a lot of batteries included, and makes it easy to get up and running with a python package.

I also had a look at the official Python Packaging User Guide, maintained by the Python Packaging Authority (PyPA). Now, I've seen that many python projects have been using the setuptools way of packaging Python modules using a setup() function within a setup.py script. Even the cookiecutter template I was using had this setup. However, it is now recommended to use the PEP 517 approach – a pyproject.toml file and a setup.cfg file. This is the approach I settled for.

At this point, I was already familiar with pyproject.toml, having already used in in my projects to define configurations for black and commitizen-tools. However, using setup.cfg was new for me, but by reading the docs and seeing how other python projects used it, I was able to figure things out. You can have a look at the project.toml and setup.cfg files on the project's GitHub page.

I used PyPA build to build my package and generate a distribution:

python -m build

This created a dist directory in the project root, with two files:

❯ ls dist/
    readme_coverage_badger-0.1.0-py3-none-any.whl 
    readme-coverage-badger-0.1.0.tar.gz

I found the following resources very helpful:

• https://setuptools.readthedocs.io/en/latest/build_meta.html
• https://packaging.python.org/key_projects/
• https://github.com/pyscaffold/pyscaffold/blob/master/setup.cfg

Some key takeaways:

• Previously, I would have multiple python configurations in my project root; one for Pytest, one for Flake8, one for Coverage and then of course pyproject.toml for black and commitizen-tools. From this experience, I realized that I could ditch the former three config files and have the configurations defined in setup.cfg! This is great, now I can reduce the number of config files in my project!
• I discovered that there are also other third-party tools one could use to package your python projects. One of the prominent ones is flit, which is used on the FastAPI project. Poetry uses project.toml and has building and packaging capabilities.

Supporting multiple Python versions

Up to this point, most of my CI/CD workflows focused on one Python version. If I'm building something that I know will be used on a Python 3.8 server, why should I bother testing it on other Python versions? Well, if you're building something that will be used by not only yourself but also others and in different environments, you have to at least ensure that it actually works in different environments. Most Python projects these days seem to support at least Python 3.6 going up, so I also adopted this approach. For the first time, I learnt how to configure and use tox in my project.

I already had pyenv installed on my machine, with one or two python versions. I made sure I had Python 3.6 to 3.9 installed, and created a .python-version file in my project root with the details of the python versions I installed:

3.6.10
3.7.9
3.8.6
3.9.0

It was such a magical experience to see tox do its thing! Of course things didn't work right the first time, for instance, it took a couple of failures and Google searches to learn that I had to create the above .python-version file!

The cookiecutter template comes with a travis.yml config which uses tox-travis for seamless integration of tox into Travis CI. Even though different Python versions are covered, all of these are on a Linux machine. I wanted to setup a test suite that also runs on Windows and Mac OS. However, my initial attempts were unsuccessful and I temporarily gave up on this, incurring technical debt in the process (it's one of my TODO items on the project)!

Key takeaway:

• I learnt how to use tox, a very powerful tool which, among other things, helps in checking that your package installs correctly with different Python versions and interpreters.

Uploading to TestPyPI, then PyPI

Before officially releasing your package to PyPI, it is recommended that you first test it on TestPyPI. After creating an account, I was able to upload my package to TestPyPI using twine:

twine upload -r testpypi dist/*

Running the above command uploads everything in your dist directory to TestPyPI. You'll be prompted for your username and password. In order to avoid the username/password prompts, you can define your package indexes configuration in a .pypirc file in your home directory.

I was able to see my project immediately on https://test.pypi.org, and even install as follows:

pip install -i https://test.pypi.org/simple/ readme-coverage-badger

In addition, I was able to

• see how the package's README is rendered,
• see whether I had correctly set the trove classifiers for the package.

As of June 2021, markdown checkboxes are not rendered at all on PyPI. They instead show up as a bullet followed by <input type="checkbox" disabled="" />, which is so annoying. Perhaps this could be one of the reasons why a number of packages maintain the use of restructuredtext for all forms of documentation. Even the README, CONTRIBUTING, AUTHORS and HISTORY files generated by audreyfeldroy/cookiecutter-pypackage are in restructuredtext format. Well, as of July 2021, I am more comfortable writing in markdown (Isn't this the same reason mkdocs was born, and why Material for MkDocs has gained popularity?). So, for now, it's README.md for me, and not README.rst!

After making corrections and rebuilding the package, I was ready to publish to the actual PyPI. So I went ahead and created an account at PyPI, then I uploaded my package:

twine upload dist/*

Side note:

The silly thing is that, while I spent time checking "secondary" things like text rendering, spellings, trove classifiers, etc., I didn't actually run my application after installing it from TestPyPI! I was comfortable with the fact that I had executed the application before building it, that I had written tests which passed and that I was able to install the app without encountering errors! But I never ran the application after building it! I only decided to do this after I had already pushed version 0.1.0 to PyPI, and to my shame, the app was broken:

ERROR: 06-Jul-21 22:02:03 There's no README.md or README at this location
INFO: 06-Jul-21 22:02:03 Run this in a directory containing either a README.md or README file

Wait a minute? What do you mean "there's no README.md ..."? I double-checked, and there was a README.md in the current directory. I went back to the code and discovered that the problem was in the way I defined the README location. I initially defined it using relative paths, and if you run the code from the project root, it works! However, this changes when you build the application, due to the notion of entry points. So, instead of using relative paths to define the README location, I used os.getcwd(), so that the script checks for the README file in the current directory, rather than checking for it in the script's directory's parent directory:

Buggy code:

def readme_location(filename: Union[str, str] = "README.md") -> Path:
    """Path to the README file"""
    current_dir = Path(__file__).resolve().parent
    parent_dir = current_dir.parents[0]
    readme_file = parent_dir / filename

    return readme_file

Fixed code:

def readme_location(filename: Union[str, str] = "README.md") -> str:
    """Path to the README file"""
    current_dir = os.getcwd()
    readme_file = os.path.join(current_dir, filename)

    return readme_file

Isn't the fixed code simpler and easier to read than the buggy one? Clearly, simple is better than complex!

After the bugfix and comprehensive testing to make sure that it actually worked, I released version 0.1.1, which is the current version at the time of writing this post.

Automating PyPI deployment

The Travis CI config that comes with the cookiecutter allows for automatic release to PyPI when you push a new tag to master. At this point, I was not very familiar with Travis CI, so I had to learn how to use it and configure it. The relevant section of the .travis.yml config is this one:

deploy:
  provider: pypi
  distributions: sdist bdist_wheel
  user: {{ cookiecutter.pypi_username }}
  password:
    secure: PLEASE_REPLACE_ME
  on:
    tags: true
    repo: {{ cookiecutter.github_username }}/{{ cookiecutter.project_slug }}
    python: 3.8

According to PyPI:

API tokens provide an alternative way (instead of username and password) to authenticate when uploading packages to PyPI

So I created an API token for the project, and set the username (user in the above config) to __token__ as per PyPI instructions. However, instead of putting the actual token in my config, I had to encrypt it:

# first, ensure that you have the travis Ruby gem installed on your computer 
# see <https://github.com/travis-ci/travis.rb#installation> for detailed installation instructions
gem install travis --no-document
# then, you need to create a [personal access token](https://github.com/settings/tokens) on Github
# following the [Travis CI guidelines](https://docs.travis-ci.com/user/github-oauth-scopes/)
# you can now sign in to Travis CI
travis login --pro --github-token YOUR_GITHUB_PERSONAL_ACCESS_TOKEN
# now you can encrypt your PyPI API token
# the `--add deploy.password` option automatically updates the `travis.yml` with the encrypted string
travis encrypt YOUR_PYPI_API_TOKEN --add deploy.password --com

The good thing is, you can easily modify this configuration to upload to TestPyPI instead. This is done by adding server: https://test.pypi.org/legacy/ under the deploy section of the config.

I modified the config slightly by adding a before_deploy section in the Travis config, which run python -m build, just as I would if I was deploying manually from my computer

Automating GitHub releases

I decided to take things further and also automate the process of creating GitHub releases. Now, there are many ways to do this, including the use of GitHub Actions. However, since I was already using Travis CI, I decided to adopt a Travis solution, since Travis CI already comes with built-in deployment support for several providers.

The GitHub releases are tied to the PyPI deployment process. So, immediately after deploying to PyPI, I wanted to create a GitHub release. Travis makes it easy to upload to multiple providers. There's even a dedicated page in the Travis docs for each provider. It is highly recommended to use travis setup releases to automatically create and encrypts a GitHub OAuth token with the correct scopes. However, this somehow didn't work for me, (I also tried adding --com and --com --force but to no avail) so I went with the manual approach.

In order for automatic GitHub releases to work,

• I automated CHANGELOG generation using commitizen-tools and standard-version.
• I created an invoke task called get-release-notes to extract the latest content from the CHANGELOG, which is basically everything under the current tag. get-release-notes saves this content to a file named LATEST_RELEASE_NOTES.md in the home directory.
• I added invoke get-release-notes to the before_deploy section of the Travis config.

The complete .travis.yml file looks like this:

language: python
os: linux
dist: focal
python:
- 3.9
- 3.8
- 3.7
- 3.6
install:
- pip install -U tox-travis
- pip install codecov
script: tox
after_success:
- codecov
before_deploy:
- pip install -r requirements_dev.txt
- invoke dist
- invoke get-release-notes
- export TODAY="($(TZ=Africa/Lusaka date --iso))"
- export RELEASE_NAME="$TRAVIS_TAG $TODAY"
deploy:
  - provider: pypi
    distributions: sdist bdist_wheel
    user: __token__
    password:
      secure: ...
    on:
      tags: true
      repo: engineervix/readme-coverage-badger
      python: 3.8
    skip_existing: true
    skip-cleanup: true
  - provider: releases
    api_key:
      secure: "..."
    file_glob: true
    file: dist/*
    skip_cleanup: true
    edge: true
    name: $RELEASE_NAME
    release_notes_file: "$HOME/LATEST_RELEASE_NOTES.md"
    on:
      tags: true
      repo: engineervix/readme-coverage-badger
      python: 3.8

What next?

Well, this was quite an interesting experience, I certainly learnt a lot of new things, and I'm still learning and improving. Now that I've released my package and it's out there, I have to maintain it! One of the important maintenance tasks is to deal with security vulnerabilities and ensure that dependencies are up to date. I previously used Dependabot as the primary tool for this, but have since switched to renovate, not only because of the auto-merge feature (which GitHub's native Dependabot doesn't have, as of July 2021), but also because of the greater flexibility in terms of configuration.

I have given myself (and whoever wants to contribute!) additional challenges to improve the package, and I hope I can work on these and continue improving it.