halting problem :: (New) Adventures in CI

One of the great advantages of moving the code hosting in GNOME to GitLab is the ability to run per-project, per-branch, and per-merge request continuous integration pipelines. While we’ve had a CI pipeline for the whole of GNOME since 2012, it is limited to the master branch of everything, so it only helps catching build issues post-merge. Additionally, we haven’t been able to run test suites on Continuous since early 2016.

Being able to run your test suite is, of course, great—assuming you do have a test suite, and you’re good at keeping it working; gating all merge requests on whether your CI pipeline passes or fails is incredibly powerful, as it not only keeps your from unknowingly merging broken code, but it also nudges you in the direction of never pushing commits to the master branch. The downside is that it lacks nuance; if your test suite is composed of hundreds of tests you need a way to know at a glance which ones failed. Going through the job log is kind of crude, and it’s easy to miss things.

Luckily for us, GitLab has the ability to create a cover report for your test suite results, and present it on the merge request summary, if you generate an XML report and tell the CI machinery where you put it:

  artifacts:
    reports:
      junit:
        - "${CI_PROJECT_DIR}/_build/report.xml"

Sadly, the XML format chosen by GitLab is the one generated by JUnit, and we aren’t really writing Java classes. The JUnit XML format is woefully underdocumented, with only an unofficial breakdown of the entities and structure available. On top of that, since JUnit’s XML format is undocumented, GitLab has its own quirks in how it parses it.

Okay, assuming we have nailed down the output, how about the input? Since we’re using Meson on various projects, we can rely on machine parseable logs for the test suite log. Unfortunately, Meson currently outputs something that is not really valid JSON—you have to break the log into separate lines, and parse each line into a JSON object, which is somewhat less than optimal. Hopefully future versions of Meson will generate an actual JSON file, and reduce the overhead in the tooling consuming Meson files.

Nevertheless, after an afternoon of figuring out Meson’s output, and reverse engineering the JUnit XML format and the GitLab JUnit parser, I managed to write a simple script that translates Meson’s testlog.json file into a JUnit XML report that you can use with GitLab after you ran the test suite in your CI pipeline. For instance, this is what GTK does:

set +e

xvfb-run -a -s "-screen 0 1024x768x24" \
    meson test \
         -C _build \
         --timeout-multiplier 2 \
     --print-errorlogs \
     --suite=gtk \
     --no-suite=gtk:gsk \
     --no-suite=gtk:a11y

# Save the exit code, so we can reuse it
# later to pass/fail the job
exit_code=$?

# We always run the report generator, even
# if the tests failed
$srcdir/.gitlab-ci/meson-junit-report.py \
        --project-name=gtk \
        --job-id="${CI_JOB_NAME}" \
        --output=_build/${CI_JOB_NAME}-report.xml \
        _build/meson-logs/testlog.json

exit $exit_code

Which results in this:

The JUnit cover report in GitLab is only shown inside the merge request summary, so it’s not entirely useful if you’re developing in a branch without opening an MR immediately after you push to the repository. I prefer working on feature branches and getting the CI to run on my changes without necessarily having to care about opening the MR until my work is ready for review—especially since GitLab is not a speed demon when it comes to MRs with lots of rebases/fixup commits in them. Having a summary of the test suite results in that case is still useful, so I wrote a small conversion script that takes the testlog.json and turns it into an HTML page, with a bit of Jinja templating thrown into it to avoid hardcoding the whole thing into string chunks. Like the JUnit generator above, we can call the HTML generator right after running the test suite:

$srcdir/.gitlab-ci/meson-html-report.py \
        --project-name=GTK \
        --job-id="${CI_JOB_NAME}" \
        --output=_build/${CI_JOB_NAME}-report.html \
        _build/meson-logs/testlog.json

Then, we take the HTML file and store it as an artifact:

  artifacts:
    when: always
    paths:
      - "${CI_PROJECT_DIR}/_build/${CI_JOB_NAME}-report.html"

And GitLab will store it for us, so that we can download it or view it in the web UI.

There are additional improvements that can be made. For instance, the reftests test suite in GTK generates images, and we’re already uploading them as artifacts; since the image names are stable and determined by the test name, we can create a link to them in the HTML report itself, so we can show the result of the failed tests. With some more fancy HTML, CSS, and JavaScript, we could have a nicer output, with collapsible sections hiding the full console log. If we had a place to upload test results from multiple pipelines, we could even graph the trends in the test suite on a particular branch, and track our improvements.

All of this is, of course, not incredibly novel; nevertheless, the network effect of having a build system in Meson that lends itself to integration with additional tooling, and a code hosting infrastructure with native CI capabilities in GitLab, allows us to achieve really cool results with minimal glue code.