halting problem :: Documentation changes

Back in the late ‘90s, people working on GTK had the exact same problem we have today: how do we document the collection of functions, types, macros, and assorted symbols that we call “an API”. It’s all well and good to strive for an API that can be immediately grasped by adhering to a set of well defined conventions and naming; but nothing is, or really can be, “self documenting”.

When GTK 1.0 was released, the documentation was literally stored in handwritten Texinfo files; the API footprint of GTK was small enough to still make it possible, but not really maintainable in the longer term. In 1998, a new system was devised for documenting GTK 1.2:

• a script, to parse the source files for the various declarations and dump them into machine parseable “templates”, that would then be modified to include the actual documentation, and committed to the source repository
• a small tool that would generate, compile, and run a small tool to introspect the type system for things like hierarchy and signals
• a script to take the templates, the list of symbols divided into logical “sections”, an index file, and generate a bunch of DocBook XML files
• finally, a script to convert DocBook to HTML or man pages, via xsltproc and an XML stylesheet

Whenever somebody added a new symbol to GTK, they would need to run the script, find the symbol in the template files, write the documentation using DocBook tags if necessary, and then commit the changes alongside the rest of the code.

Since this was 1998, and the scripts had to parse a bunch of text files using regular expressions, they were written in Perl and strung together with a bunch of Makefile rules.

Thus, gtk-doc was born.

Of course, since other libraries needed to provide an API reference to those poor souls using them, gtk-doc ended up being shared across the GNOME platform. We even built part of our website and release infrastructure around it.

At some point between 1998 and 2009, gtk-doc gained the ability to generate those template files incrementally, straight from the C sources; this allowed moving the preambles of each section into the corresponding source file, thus removing the templates from the repository, and keeping the documentation close to the code it references, in the hope it would lead to fewer instances of docs drift.

---

Between 2009 and 2021, a few things happened:

1. gobject-introspection has become “a thing”; g-i also parses the C code, and does it slightly more thoroughly than gtk-doc, to gather the same information: declarations, hierarchy, interfaces, properties, and signals, and even documentation, which is all shoved into a well-defined XML file; on top of that, g-i needs annotations in the source to produce a machine-readable description of the C ABI of a library, which can then be used to generate language bindings
2. turns out that DocBook is pretty terrible, and running xsltproc on large, complex DocBook files is really slow
3. Perl isn’t really a Hot Language™ like it was in the late ‘90s; many Linux distributions dropped it from the core installation, and not many people speak it that fluently, which means not many people will want to help with a large Perl application

To cope with issue (1), gtk-doc had to learn to parse introspection annotations.

Issue (2) led to replacing DocBook tags inside the inline documentation with subset of Markdown, augmented with custom code blocks and intra-document anchors for specific sections.

Issue (3) led to a wholesale rewrite in Python, in the hope that more people would contribute to the maintenance of gtk-doc.

Sadly, all three solutions ended up breaking things in different ways:

1. gtk-doc never really managed to express the introspection information in the generated documentation, outside of references to an ancillary appendix. If an annotation says “this argument can be NULL”, for instance, there’s no need to write “or NULL” in the documentation itself: the documentation tool can write it out for you.
2. the move to Markdown means that existing DocBook tags in the documentation are now ignored or, worse, misinterpreted for HTML and not rendered; this requires porting all the documentation in every library, in a giant flag day, to avoid broken docs; on top of that, DocBook’s style sheet to generate HTML started exhibiting regressions after a build system change, which led, among other things, to the disappearance of per-version symbols indices
3. the port to Python probably came too late, and ended up having many, many regressions; gtk-doc is still a pretty complex tool, and it still caters to many different use cases, spanning two decades; as much as its use is documented and tested, its internals are really not, meaning that it’s not an easy project to pick up

Over the past 10 years various projects started migrating away from gtk-doc; gobject-introspection itself shipped a documentation tool capable of generating API references, though it mostly is a demonstrator of potential capabilities more than an actual tool. Language bindings, on the other hand, adopted the introspection data as the source for their documentation, and you can see it in Python, JavaScript, and Rust.

As much as I’d like to contribute to gtk-doc, I’m afraid we reached the point where we might want to experiment with something more radical, instead of patching something up, and end up breaking what’s left.

So, since we’re starting from the bottom up, let’s figure out what are the requirements for a tool to generate the documentation for GTK:

• be fast. Building GTK’s API reference takes a long time. The API footprint of GDK, GSK, and GTK is not small, but there’s no reason why building the documentation should take a comparable amount of time as building the library. We moved to Meson because it has improved the build times of GTK, we don’t want to get into a bottleneck now.
• no additional source parsing. We already parse the C sources in order to generate the introspection data, we don’t need another pass at that.
• tailored for GTK. Whenever GTK changes, the tool must change with GTK; the output must adapt to the style of documentation GTK uses.
• integrated with GTK. We don’t want an external dependency that makes it harder to deploy the GTK documentation on non-Linux platforms. Using it as a sub-project would be the best option, followed by being able to install it everywhere without additional, Linux-only dependencies.

The explicit non-goal is to create a general purpose documentation tool. We don’t need that; in fact: we’re actively avoiding it. Regardless of what you’ve been taught at university, or your geeky instincts tell you, not every problem requires a generic solution. The whole reason why we are in this mess is that we took a tool for generating the GTK documentation and then generalised the approach until it fell apart under its own weight.

If you want a general purpose documentation tool for C and C++ libraries, there are many to choose from:

• Doxygen
• HotDoc
• MarkDeep

There’s also gtk-doc: if you’re using it already, I strongly recommend helping out with its maintenance.

---

Back in November 2020, as a side project while we were closing in to the GTK 4.0 release date, I started exploring the idea of parsing the introspection data to generate the C API reference for GTK. I wanted to start from scratch, and see how far I could go, so I deliberately avoided taking the GIR parser from gobject-introspection; armed only with the GIR schema and a bunch of Python, I ended up writing a decent parser that would be able to load the GTK introspection XML data, including its dependencies, and dump the whole tree of C identifiers and symbols. After a break, at the end of January 2021, I decided to take a page out the static website generator rule book, and plugged the Jinja templates into the introspection data. The whole thing took about a couple of weeks to go from this:

to this:

My evil plan of generating something decent enough to be usable and then showing it to people with actual taste and web development skills paid off, because I got a whole merge request from Martin Zilz to create a beautiful theme, with support for responsive layout and even for a dark variant:

Turns out that when you stop parsing C files and building small binaries to introspect the type system, and remove DocBook and xsltproc from the pipeline, things get fast. Who knew…

Additionally, once you move the template and the styling outside of the generator, and you can create more complex documentation hierarchies, while retaining the ability for people that are not programmers to change the resulting HTML.

The interesting side effect of using introspection data is that our API reference is now matching what language bindings are able to see and consume—and oh boy, do we suck at that. Part of the fault lies in the introspection parser not being able to cover some of the nastiest parts of C, like macros—though, hopefully, that will improve in the near future; but a lot of issues come from our own API design. Even after 10 years since the introduction of introspection, we’re still doing some very dumb things when it comes to C API—ad let’s ignore stuff that happened 20 years ago and that we haven’t been able to fix yet. Hopefully, the documentation slapping us in the face is going to help us figuring things out before they hit stable releases.

---

What’s missing from gi-docgen? Well, you can look at the 2021.1 milestone on GitLab:

• more documentation on the ancillary files used for project and template configuration; stabilising the key/value pairs would also be part of the documentation effort
• client-side search, with symbols exposed to tools like GNOME Builder through something that isn’t quite as tragic as DevHelp files
• automatic cross-linking with dependencies documented by gi-docgen, especially for libraries with multiple namespaces, like GTK and Pango
• generating proper dependency files, for the consumption of build tools like Meson

In the meantime, what’s missing for GTK to use this? Mainly, porting the documentation away from the various gtk-doc-isms, like marking symbols with sigils, or using |[ ... ]| to define code blocks. Additionally, since the introspection scanner only attaches SECTION blocks to the documentation element of a class, all the sections that operate as “grab bag of related symbols” need to be moved to a separate Markdown file.

It must needs be remarked: gi-docgen is not a generic solution for documenting C libraries. If your library does not have introspection, doesn’t use type classes, or has a very different ABI exposed through introspection than the actual C API, then you’re not going to find it useful—and I don’t have any plans to cater to your use cases either. You should keep using gtk-doc if it still works for you; or you may want to consider other documentation tools.

Anything that complicates the goal of this tool—generating the API reference for GTK and ancillary libraries—is completely out of scope.

---

• The gi-docgen repository
• A demo render of the GTK reference
• The MR for GTK