halting problem :: Publishing your documentation

:: ~4 min read

The main function of library-web, the tool that published the API reference of the various GNOME libraries, was to take release archives and put their contents in a location that would be visible to a web server. In 2006, this was the apex of automation, of course. These days? Not so much.

Since library-web is going the way of the Dodo, and we do have better ways to automate the build and publishing of files with GitLab, how do we replace library-web in 2021? The answer is, unsurprisingly: continuous integration pipelines.

I will assume that you’re already building—and testing—your library using GitLab’s CI; if you aren’t, then you have bigger problems than just publishing your API.

So, let’s start with these preconditions:

If your project doesn’t satisfy these preconditions you might want to work on doing so; alternatively, you can implement your own CI pipeline.

Let’s start with a simple job template:

# Expected variables:
# PROJECT_DEPS: the dependencies for your own project
# MESON_VERSION: the version of Meson you depend on
# MESON_EXTRA_FLAGS: additional Meson setup options
#   you wish to pass to the configuration phase
# DOCS_FLAGS: the Meson setup option for enabling the
#   documentation, if any
# DOCS_PATH: the path of the generated reference,
#   relative to the build root
.gidocgen-build:
  image: fedora:latest
  before_script:
    - export PATH="$HOME/.local/bin:$PATH"
    - >
      dnf install -y
      python3
      python3-pip
      python3-wheel
      gobject-introspection-devel
      graphviz
      ninja-build
      redhat-rpm-config
    - dnf install -y ${PROJECT_DEPS}
    - >
      pip3 install
      meson==${MESON_VERSION}
      gi-docgen
      jinja2
      Markdown
      markupsafe
      pygments
      toml
      typogrify
  script:
    - meson setup ${MESON_EXTRA_FLAGS} ${DOCS_FLAGS} _docs .
    - meson compile -C _docs
    - |
      pushd "_docs/${DOCS_PATH}" > /dev/null 
      tar cfJ ${CI_PROJECT_NAME}-docs.tar.xz .
      popd > /dev/null
    - mv _docs/${DOCS_PATH}/${CI_PROJECT_NAME}-docs.tar .
  artifacts:
    when: always
    name: 'Documentation'
    expose_as: 'Download the API reference'
    paths:
      - ${CI_PROJECT_NAME}-docs.tar.xz

This CI template will:

  • download all the required dependencies for building the API reference using gi-docgen
  • build your project, including the API reference
  • create an archive with the API reference
  • store the archive as a CI artefact that you can easily download

Incidentally, by adding a meson test -C _build to the script section, you can easily test your build as well; and if you have a test() target in your build that runs gi-docgen check, then you can verify that your documentation is always complete.

Now, all you have to do is create your own CI job that inherits from the template inside its own stage. I will use JSON-GLib as a reference:

stages:
  - docs

api-reference:
  stage: docs
  extends: .gidocgen-build
  needs: []
  variables:
    MESON_VERSION: 0.55.3
    DOCS_FLAGS: -Dgtk_doc=true
    PROJECT_DEPS:
      gcc
      git
      gettext
      glib2-devel
    DOCS_PATH: docs/json-glib-1.0

What about gtk-doc!”, I hear from the back of the room. Well, fear not, because there’s a similar template you can use if you’re still using gtk-doc in your project:

# Expected variables:
# PROJECT_DEPS: the dependencies for your own project
# MESON_VERSION: the version of Meson you depend on
# MESON_EXTRA_FLAGS: additional Meson setup options you
#   wish to pass to the configuration phase
# DOCS_FLAGS: the Meson setup option for enabling the
#   documentation, if any
# DOCS_TARGET: the Meson target for building the
#   documentation, if any
# DOCS_PATH: the path of the generated reference,
#   relative to the build root
.gtkdoc-build:
  image: fedora:latest
  before_script:
    - export PATH="$HOME/.local/bin:$PATH"
    - >
      dnf install -y
      python3
      python3-pip
      python3-wheel
      gtk-doc
      ninja-build
      redhat-rpm-config
    - dnf install -y ${PROJECT_DEPS}
    - pip3 install  meson==${MESON_VERSION}
  script:
    - meson setup ${MESON_EXTRA_FLAGS} ${DOCS_FLAGS} _docs .
    # This is exceedingly annoying, but sadly its how
    # gtk-doc works in Meson
    - ninja -C _docs ${DOCS_TARGET}
    - |
      pushd "_docs/${DOCS_PATH}" > /dev/null 
      tar cfJ ${CI_PROJECT_NAME}-docs.tar.xz .
      popd > /dev/null
    - mv _docs/${DOCS_PATH}/${CI_PROJECT_NAME}-docs.tar .
  artifacts:
    when: always
    name: 'Documentation'
    expose_as: 'Download the API reference'
    paths:
      - ${CI_PROJECT_NAME}-docs.tar.xz

And now you can use extends: .gtkdoc-build in your api-reference job.

Of course, this is just half of the job: the actual goal is to publish the documentation using GitLab’s Pages. For that, you will need another CI job in your pipeline, this time using the deploy stage:

stages:
  - docs
  - deploy

# ... the api-reference job goes here...

pages:
  stage: deploy
  needs: ['api-reference']
  script:
    - mkdir public && cd public
    - tar xfJ ../${CI_PROJECT_NAME}-docs.tar.xz
  artifacts:
    paths:
      - public
  only:
    - master
    - main

Now, once you push to your main development branch, your API reference will be built by your CI pipeline, and the results published in your project’s Pages space—like JSON-GLib.

The CI pipeline and GitLab Pages are also useful for building complex, static websites presenting multiple versions of the documentation; or presenting multiple libraries. An example of the former is libadwaita’s website, while an example of the latter is the GTK documentation website. I’ll write a blog post about them another time.


Given that the CI templates are pretty generic, I’m working on adding them into the GNOME ci-templates repository, so you will be able to use something like:

include: 'https://gitlab.gnome.org/GNOME/citemplates/raw/HEAD/docs/gidocgen.yml'

or:

include: 'https://gitlab.gnome.org/GNOME/citemplates/raw/HEAD/docs/gtkdoc.yml'

without having to copy-paste the template in your own .gitlab-ci.yml file.


The obvious limitation of this approach is that you will need to depend on the latest version of Fedora to build your project. Sadly, we cannot use Flatpak and the GNOME run time images for this, mainly because we are building libraries, not applications; and because extracting files out of a Flatpak build after it has completed isn’t entirely trivial. Another side effect is that if you bump up the dependencies of your project to something on the bleeding edge and currently not packaged on the latest stable Fedora, you will need to have it included as a Meson sub-project. Of course, you should already be doing that, so it’s a minor downside.

Ideally, if GNOME built actual run time images for the SDK, we could install gtk-doc, gi-docgen, and all their dependencies into the SDK itself, and avoid depending on a real Linux distribution for the libraries in our platform.

gnome development documentation automation gitlab ci

Follow me on Mastodon