One of the great advantages of moving the code hosting in GNOME
to GitLab is the ability to run per-project, per-branch, and
per-merge request continuous integration pipelines. While we’ve had a CI
pipeline for the whole of GNOME since 2012, it is limited to
the master
branch of everything, so it only helps catching build issues
post-merge. Additionally, we haven’t been able to run test suites on
Continuous since early 2016.
Being able to run your test suite is, of course, great—assuming you do have
a test suite, and you’re good at keeping it working; gating all merge
requests on whether your CI pipeline passes or fails is incredibly powerful,
as it not only keeps your from unknowingly merging broken code, but it also
nudges you in the direction of never pushing commits to the master
branch.
The downside is that it lacks nuance; if your test suite is composed of
hundreds of tests you need a way to know at a glance which ones failed.
Going through the job log is kind of crude, and it’s easy to miss things.
Luckily for us, GitLab has the ability to create a cover report for your test suite results, and present it on the merge request summary, if you generate an XML report and tell the CI machinery where you put it:
artifacts:
reports:
junit:
- "${CI_PROJECT_DIR}/_build/report.xml"
Sadly, the XML format chosen by GitLab is the one generated by JUnit, and we aren’t really writing Java classes. The JUnit XML format is woefully underdocumented, with only an unofficial breakdown of the entities and structure available. On top of that, since JUnit’s XML format is undocumented, GitLab has its own quirks in how it parses it.
Okay, assuming we have nailed down the output, how about the input? Since we’re using Meson on various projects, we can rely on machine parseable logs for the test suite log. Unfortunately, Meson currently outputs something that is not really valid JSON—you have to break the log into separate lines, and parse each line into a JSON object, which is somewhat less than optimal. Hopefully future versions of Meson will generate an actual JSON file, and reduce the overhead in the tooling consuming Meson files.
Nevertheless, after an afternoon of figuring out Meson’s output, and reverse
engineering the JUnit XML format and the GitLab JUnit parser, I managed to
write a simple script that translates Meson’s
testlog.json
file into a JUnit XML report that you can use with GitLab
after you ran the test suite in your CI pipeline. For instance, this is what
GTK does:
set +e
xvfb-run -a -s "-screen 0 1024x768x24" \
meson test \
-C _build \
--timeout-multiplier 2 \
--print-errorlogs \
--suite=gtk \
--no-suite=gtk:gsk \
--no-suite=gtk:a11y
# Save the exit code, so we can reuse it
# later to pass/fail the job
exit_code=$?
# We always run the report generator, even
# if the tests failed
$srcdir/.gitlab-ci/meson-junit-report.py \
--project-name=gtk \
--job-id="${CI_JOB_NAME}" \
--output=_build/${CI_JOB_NAME}-report.xml \
_build/meson-logs/testlog.json
exit $exit_code
Which results in this:
The JUnit cover report in GitLab is only shown inside the merge request
summary, so it’s not entirely useful if you’re developing in a branch
without opening an MR immediately after you push to the repository. I prefer
working on feature branches and getting the CI to run on my changes without
necessarily having to care about opening the MR until my work is ready for
review—especially since GitLab is not a speed demon when it comes to MRs
with lots of rebases/fixup commits in them. Having a summary of the test
suite results in that case is still useful, so I wrote a small conversion
script that takes the testlog.json
and turns it into
an HTML page, with a bit of Jinja templating thrown into it to
avoid hardcoding the whole thing into string chunks. Like the JUnit
generator above, we can call the HTML generator right after running the test suite:
$srcdir/.gitlab-ci/meson-html-report.py \
--project-name=GTK \
--job-id="${CI_JOB_NAME}" \
--output=_build/${CI_JOB_NAME}-report.html \
_build/meson-logs/testlog.json
Then, we take the HTML file and store it as an artifact:
artifacts:
when: always
paths:
- "${CI_PROJECT_DIR}/_build/${CI_JOB_NAME}-report.html"
And GitLab will store it for us, so that we can download it or view it in the web UI.
There are additional improvements that can be made. For instance, the reftests test suite in GTK generates images, and we’re already uploading them as artifacts; since the image names are stable and determined by the test name, we can create a link to them in the HTML report itself, so we can show the result of the failed tests. With some more fancy HTML, CSS, and JavaScript, we could have a nicer output, with collapsible sections hiding the full console log. If we had a place to upload test results from multiple pipelines, we could even graph the trends in the test suite on a particular branch, and track our improvements.
All of this is, of course, not incredibly novel; nevertheless, the network effect of having a build system in Meson that lends itself to integration with additional tooling, and a code hosting infrastructure with native CI capabilities in GitLab, allows us to achieve really cool results with minimal glue code.