Integration and Delivery

Writing an application can be a fun pastime, but creating a working (web-)application calls for careful integration and delivery to production sites. You wouldn't run your web-app from your laptop to the whole wide world, would you?

To get the application to actually work outside of a programmer's development environment we need some prerequisites (depending on the application itself) and some sort of deployment environment.

TL;DR We set up packaging, service and operating system in a dedicated git repository handy for reproduction, (continuous) integration and delivery.

The prerequisites

VPS

Of course, a machine of sorts is needed. The easiest and cheapest solution is to rent a Virtual Private Server by any one of the existing cloud server providers.

DNS

For the application to be reachable through the interwebs, we need to buy a domain name and configure some nameservers (the easiest way is to use the ones usually provided by the domain registrar) to resolve our domain with the IP address(es) of our host.

Since I am trying to prove some points here - and of course learn some valuable lessons by the way - I opt to authorize my own domain root zone and resolve my names and IP addresses myself.

Email

Since the very core of the application is to register (valid) email addresses of people trying to download the files, we need means to send email. The easy way would be to use credentials for an SMTP server of some email provider, but this would not be half as much fun - and not teach me any lessons.

The package

Of course, for building and deployment our software needs packaging. It is one of the main differences (or you could argue: advantages) that GNU Guix opts for a full-fledged, well established, general-purpose programming language for both code and configuration of all sorts. The result is a simple-looking, well-defined, easily machine-readable package definition in just 33 lines of Guile scheme.

(define-public hello
  (package
    (name "hello")
    (version "2.12.1")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "086vqwk2wl8zfs47sq2xpjc9k066ilmb8z6dn0q6ymwjzlm196cd"))))
    (build-system gnu-build-system)
    (synopsis "Example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))

The service(s)

From the server perspective, any program that is supposed to be kept alive is referred to as a service. Since there is no general service to run Django apps in GNU Guix (yet) we have to craft our own - leaving an excellent exercise to showcase what Guix' internals look like.

On legacy systems services need be installed, often through some sort of magic in the installation process of a package that also installs systemd service files which are supposed to be correct for all systems but from time to time need some manual tweaking. To the unsuspecting system operator this may seem like hidden magic, to the curious system administrator this may look like somewhat clearly defined behavior, to the crafter of more interesting system setups things can quickly become obtuse - or at the very least in need for manual labor to get to work.

The reproducibility built into the design core of GNU Guix works in a simple manner: first we define everything necessary as directed, acyclic graphs (DAG), then we parse those graphs to generate reproducible definitions (of both software, services and operating systems). The graphs ensure us on a theoretical level - as long as there are no loops we have no circular dependencies - while the translation process does the work. So, many things are services that may seem unnecessarily so on first glance: home directory mounting as well as user management. But it makes a lot of sense! For example: having defined users as (extensions to) a service, each new service can extend this service by some new users. There is no need for someone to manually edit /etc/passwd or have some dubious, unknown script edit it for you (can you guarantee the script always works as you intend?) - we simply define it within the service and make that service part of our system definition. Our well-structured high-level programming abstractions take care of the rest, at a degree that is easily verifiable and in a code base that remains well maintainable.

With well crafted services we don't have to care whether we run one or 5 instances of a web- or database server. We have no mess with configuration files in /etc, because the configuration files themselves don't get to live there.

The variable django-deployment-service-type defines our service as an extension of the root shepherd service (to make management commands available through the herd <command> <service> interface, of the account-service-type (so we can run our service as a separate user instead of root) and the nginx-service-type, so we don't have to create dedicated nginx reverse-proxy configuration for each instance of our application.

But have a look yourself.

The (production) operating system

The new age has begun! The old days of manually flashing some ISO, running some curses wizard to install a system which then needs hours of (re-)configuration (meaning: manually copying and/or editing several configuration files until it finally works) are over.

In this new age whole operating systems are declared in a holistic manner - at the same time as versatile as a user's actual needs and as agnostic as they could be. From hostname all the way through network configuration and system services, a Guix System is to be declared, configured onto a machine and then enjoyed. If something fails we simply roll it back. If anything needs adjustment, we simply edit the declaration and reconfigure the system. And of course, we keep track of our changes through with git.

Since I went a bit overboard and did not just create one single machine serving the web-application, I could use this extra mile to demonstrate just how practical it is to declare with a full-fledged, modern programming language.

Since domain name authorities need two equally configured domain name servers, I first declare a minimal subset of the operating system with all the essential services configured: DHCP, nftables, ntp, unattended upgrades, knot (DNS) and SSH. These are stored in the (module-local) variable %services. I then define a minimal operating system %base-system which the name-server-only operating system inherits and extends only by its hostname.

The actual web-application serving operating-system %system also extends %base-system but adds all the necessary services: openSMTP, certbot, nginx and our hand-crafted Django deployment service.

The advantages should be obvious: changing the layout of a single-machine deployment to a multi-machine only takes altering the system definitions and deploying these to the adequate machines. Making good use of the tools GNU Guile provides we can ensure integrity of these systems from within our code base. This could encompass: the correct configuration of all the systems to use the right IP address for the dedicated database-server, use the same NFS mount-points for network backups, install the same and correct public keys for the relevant users, etc. The options are only limited by the system designer's fantasy.

Integration and Delivery

Putting it all together we are able to test our setups locally by building the application software as a package or the operating system(s) as a VM image (or docker container, if you prefer) and test on a development machine. We are able to deploy these definitions to our VPSs (remember the machine.scm file) and have our operating systems pull the most recent versions through unattended upgrade mechanisms (continuous delivery, if you will).

All in all, we are enabled to keep a clean, tidy, well maintainable bundle of software, service and operating system definitions that are as expressive as they are reproducible.