halting problem :: Episode 2.a: Building GNOME

:: ~8 min read

A side episode! Building GNOME is complicated; releasing GNOME is even worse. We’re going to see what tools GNOME developers used to build GNOME 2.

In the GNOME project community, the people building the code are represented by two separate but equally important groups: the maintainers, who release the code, and the release team, who release GNOME. These are their stories.


Developing a software project can be hard, but building one ought to be simpler, right? After all, it’s kind of a binary state for software projects after any change: either the change broke the build, or it didn’t—and broken builds do not get released, right?

Oh, you sweet summer child. Of course it’s not that simple.

Building software gets even more complicated when it comes to complex, interdependent projects composed by multiple modules like GNOME; each module has dependencies lower in the stack, and reverse dependencies—that is, modules that depend on the interfaces it provides—higher in the stack.

If you’re working on a component low in the stack, especially in 2002, chances are you only have system dependencies, and those system dependencies are generally shipped by your Linux distribution. Those dependencies do not typically change very often, or at the very least they assume you’re okay with not targeting the latest, bleeding edge version. If push comes to shove, you can always write a bunch of fallback code that gets only ever tested by people running on old operating systems—if they decide on a whim to try and compile the latest and greatest application, instead of just waiting until their whole platform gets an upgrade.

Moving upwards in the GNOME stack, you start having multiple dependencies, but generally speaking those dependencies move at the same speed as your project, so you can keep track of them. Unlike other projects, the GNOME platform never had issues with the multiplication of dependencies—something that will come back to bite the maintainers later in the 2.x development cycle—but even so, with GNOME offering code hosting to like-minded developers, it didn’t use to be hard to keep track of things; if everything happens on the same source code repository infrastructure you can subscribe to the changes for your dependencies, and see what happens almost in real time.

Applications, finally, are another beast entirely; here, dependencies can be many, and spanning across different system services, different build systems, different code hosting services, and different languages. To minimise the causes of headaches, you can decide to target the versions packages by the Linux distribution you’re using, unless you’re in the middle of a major API version shift in the platform—like, say, the one that happened between GNOME 1 and 2. No Linux distribution packager in their right mind would ship releases for highly unstable core platform libraries, especially when the development cadence of those libraries outpaces the cadence of the distribution packaging and update process. For those cases, using snapshots of the source under revision control is the only option left for you as an upstream maintainer.

So, at this point, your options are limited. You want to install the latest and greatest version of your dependencies—unstable as they might be—on your local system, but you don’t want to mess up the rest of the system in case the changes introduce a bug. In other words: if you’re running GNOME as your desktop, and you upgrade a dependency for your application, you might end up breaking the your own desktop, and then spend time undoing the unholy mess you made, instead of hacking on your own project.

This is usually the point where programmers start writing scripts to modify the environment, and build dependencies and applications into their own separate prefix, using small utilities that describe where libraries put their header files and shared objects in order to construct the necessary compiler and linker arguments. GNOME libraries standardised to one of these tools, called pkg-config, before the 2.0 release, replacing the per-project tools like glib-config or gtk-config that were common during the GNOME 1.x era.

Little by little, the scripts each developer used started to get shared, and improved; for instance, instead of hard coding the list of things to build, and the order in which they should be built, you may want to be able to describe a whole project in terms of a list of modules to be built—each module pointing to its source code repository, its configuration and build time options, and its dependencies. Another useful feature is the ability to set up an environment capable of building and running applications against the components you just built.

The most successful script for building and running GNOME components, jhbuild, emerged in 2002. Written by James Henstridge, the maintainer of the GTK bindings for Python, jhbuild took an XML description of a set of modules, built the dependency tree for each component, and went through the set in the right order, building everything it could, assuming it knew how to handle the module’s build system—which, at the time, was mostly a choice between Autotools and Autotools. Additionally, it could spawn a shell and let you run what you built, or compile additional code as if you installed every component in a system location. If you wanted to experience GNOME at its most bleeding edge, you could build a whole module set into a system prefix like /opt, and point your session manager to that location. Running a whole desktop environment out of a CVS snapshot: what could possibly go wrong, right? Well, at least you had the option of going back to the safe harbours of your Linux distibution’s packaged version of GNOME.

Little by little, over the years, jhbuild acquire new features, like the ability to build projects that were not using Autotools. Multiple module sets, one for each branch of GNOME, and one for tracking the latest and greatest, appeared over the years, as well as additional module sets for building applications, both hosted on gnome.org repositories and outside the GNOME infrastructure. Jhbuild was even used on non-Linux platforms, like macOS, to build the core GNOME platform stack, and let application developers port their work there. Additionally, other projects like X11 and the freedesktop.org stack that we’ll see in a future episode, will publish their own module sets, as many of the developers in GNOME moved through the stack and brought their tools with them.

With jhbuild consuming sets of modules in order to build the GNOME stack, the question becomes: who maintained those sets? Ideally, the release team would be responsible for keeping the modules up to date whenever a new dependency was added, or an old dependency removed. As the release team was responsible of deciding which modules belonged in the GNOME release, they would be the ones best positioned to update the jhbuild sets. There was a small snag in this plan, though: the release team already had its own tool for building GNOME from release archives produced and published by the module maintainers, in the correct order, to verify that the whole GNOME release would build and produce something that could be packaged by Linux (and non-Linux) distributors.

The release team’s tool was called GARNOME, and was based on the GAR architecture designed by Nick Moffitt, which itself was largely based on the BSD port system. GARNOME was developed as a way for the release team to build and test alpha releases of GNOME during the 2.0 development cycle. The main difference between jhbuild and GARNOME was the latter’s focus on release archives, compared to the former’s focus on source checkouts. The main goal of GARNOME was really to replicate the process of distributors taking various releases and packaging them up in their preferred format. While editing jhbuild’s module sets was a simple matter of changing some XML, the GARNOME recipes were a fairly complicated set of Make files, with magic variables and magic include directives that would lead to building and installing each module, using Make rules to determine the dependencies. All of this meant that there was not only no overlap between who used jhbuild and who used GARNOME, but also no overlap between who contributed to which project.

Both jhbuild and GARNOME assumed you had a working system and development toolchain for all the programming languages needed to build GNOME components; they also relied on a whole host of system dependencies, especially when it came to talking to system services and hardware devices. While this was relatively less important for GARNOME, whose role was simply to build the whole of GNOME, jhbuild started to suffer from these limitations as soon as GNOME projects began interacting much more with the underlying services offered by the operating system.

It’s important to note that none of this stuff was automated; it all relied on human intervention for testing that things would not blow up in interesting ways. We were far, far away from any concept of a continuous integration pipeline. Individual developers had to hunt down breakage in library releases that would have repercussions down the line when building other libraries, system components, or applications. The net result was that building GNOME was only possible if everything was built out of release archives; anything else was deeply unstable, and proved to be hard to handle for both seasoned developers and new contributors alike, the more complexity was piled on the project.

GARNOME was pretty successful, and ended up being used for a majority of the GNOME 2 releases, until it was finally retired in favour of jhbuild itself, using a special module set that pointed to release archives instead of source code repositories. The module set was maintained by the release team, and published for every GNOME release, to let other developers and packagers reproduce and validate the process.

Jhbuild is still used to this day, mostly for the development of system components like GTK, or the GNOME Shell; application building has largely shifted towards containerised systems, like Flatpak, which have the advantage of being easily automated in a CI environment. These systems are also much easier to use from a newcomer perspective, and are quite more reliable when it comes to the stability of the underlying middleware.

The release team switched away from jhbuild for validating and publishing GNOME releases in 2018, long into the GNOME 3 release cycle, using a new tool called BuildStream which not only builds the GNOME components but it also builds the lower layers of the stack including the compiler toolchain, to ensure a level of build reproducibility that jhbuild and GARNOME never had.

References

history of gnome gnome podcast

Follow me on Mastodon