halting problem - gnome

Let’s talk about Moonforge

2026-03-17T17:45:00+00:00

Last week, Igalia finally announced Moonforge, a project we’ve been working on for basically all of 2025. It’s been quite the rollercoaster, and the announcement hit various news outlets, so I guess now is as good a time as any to talk a bit about what Moonforge is, its goal, and its constraints.

Of course, as soon as somebody announces a new Linux-based OS, folks immediately think it’s a new general purpose Linux distribution, as that’s the square shaped hole where everything OS-related ends up. So, first things first, let’s get a couple of things out of the way about Moonforge:

Moonforge is not a general purpose Linux distribution
Moonforge is not an embedded Linux distribution

What is Moonforge

Moonforge is a set of feature-based, well-maintained layers for Yocto, that allows you to assemble your own OS for embedded devices, or single-application environments, with specific emphasys on immutable, read-only root file system OS images that are easy to deploy and update, through tight integration with CI/CD pipelines.

Why?

Creating a whole new OS image out of whole cloth is not as hard as it used to be; on the desktop (and devices where you control the hardware), you can reasonably get away with using existing Linux distributions, filing off the serial numbers, and removing any extant packaging mechanism; or you can rely on the containerised tech stack, and boot into it.

When it comes to embedded platforms, on the other hand, you’re still very much working on bespoke, artisanal, locally sourced, organic operating systems. A good number of device manufacturers coalesced their BSPs around the Yocto Project and OpenEmbedded, which simplifies adaptations, but you’re still supposed to build the thing mostly as a one off.

While Yocto has improved leaps and bounds over the past 15 years, putting together an OS image, especially when it comes to bundling features while keeping the overall size of the base image down, is still an exercise in artisanal knowledge.

A little detour: Poky

Twenty years ago, I moved to London to work for this little consultancy called OpenedHand. One of the projects that OpenedHand was working on was taking OpenEmbedded and providing a good set of defaults and layers, in order to create a “reference distribution” that would help people getting started with their own project. That reference was called Poky.

We had a beaver mascot before it was cool

These days, Poky exists as part of the Yocto Project, and it’s still the reference distribution for it, but since it’s part of Yocto, it has to abide to the basic constraint of the project: you still need to set up your OS using shell scripts and copy-pasting layers and recipes inside your own repository. The Yocto project is working on a setup tool to simplify those steps, but there are alternatives…

Another little detour: Kas

One alternative is kas, a tool that allows you to generate the local.conf configuration file used by bitbake through various YAML fragments exported by each layer you’re interested in, as well as additional fragments that can be used to set up customised environments.

Another feature of kas is that it can spin up the build environment inside a container, which simplifies enourmously its set up time. It avoids unadvertedly contaminating the build, and it makes it very easy to run the build on CI/CD pipelines that already rely on containers.

What Moonforge provides

Moonforge lets you create a new OS in minutes, selecting a series of features you care about from various available layers.

Each layer provides a single feature, like:

support for a specific architecture or device (QEMU x86_64, RaspberryPi)
containerisation (through Docker or Podman)
A/B updates (through RAUC, systemd-sysupdate, and more)
graphical session, using Weston
a WPE environment

Every layer comes with its own kas fragment, which describes what the layer needs to add to the project configuration in order to function.

Since every layer is isolated, we can reason about their dependencies and interactions, and we can combine them into a final, custom product.

Through various tools, including kas, we can set up a Moonforge project that generates and validates OS images as the result of a CI/CD pipeline on platforms like GitLab, GitHub, and BitBucket; OS updates are also generated as part of that pipeline, just as comprehensive CVE reports and Software Bill of Materials (SBOM) through custom Yocto recipes.

More importantly, Moonforge can act both as a reference when it comes to hardware enablement and support for BSPs; and as a reference when building applications that need to interact with specific features coming from a board.

While this is the beginning of the project, it’s already fairly usable; we are planning a lot more in this space, so keep an eye out on the repository.

Trying Moonforge out

If you want to check out Moonforge, I will point you in the direction of its tutorials, as well as the meta-derivative repository, which should give you a good overview on how Moonforge works, and how you can use it.

Governance in GNOME

2025-08-03T20:48:00+01:00

How do things happen in GNOME?

Things happen in GNOME? Could have fooled me, right?

Of course, things happen in GNOME. After all, we have been releasing every six months, on the dot, for nearly 25 years. Assuming we’re not constantly re-releasing the same source files, then we have to come to the conclusion that things change inside each project that makes GNOME, and thus things happen that involve more than one project.

So let’s roll back a bit.

GNOME’s original sin

We all know Havoc Pennington’s essay on preferences; it’s one of GNOME’s foundational texts, we refer to it pretty much constantly both inside and outside the contributors community. It has guided our decisions and taste for over 20 years. As far as foundational text goes, though, it applies to design philosophy, not to project governance.

When talking about the inception and technical direction of the GNOME project there are really two foundational texts that describe the goals of GNOME, as well as the mechanisms that are employed to achieve those goals.

The first one is, of course, Miguel’s announcement of the GNOME project itself, sent to the GTK, Guile, and (for good measure) the KDE mailing lists:

We will try to reuse the existing code for GNU programs as much as possible, while adhering to the guidelines of the project. Putting nice and consistent user interfaces over all-time favorites will be one of the projects. — Miguel de Icaza, “The GNOME Desktop project.” announcement email

Once again, everyone related to the GNOME project is (or should be) familiar with this text.

The second foundational text is not as familiar, outside of the core group of people that were around at the time. I am referring to Derek Glidden’s description of the differences between GNOME and KDE, written five years after the inception of the project. I isolated a small fragment of it:

Development strategies are generally determined by whatever light show happens to be going on at the moment, when one of the developers will leap up and scream “I WANT IT TO LOOK JUST LIKE THAT” and then straight-arm his laptop against the wall in an hallucinogenic frenzy before vomiting copiously, passing out and falling face-down in the middle of the dance floor. — Derek Glidden, “GNOME vs KDE”

What both texts have in common is subtle, but explains the origin of the project. You may not notice it immediately, but once you see it you can’t unsee it: it’s the over-reliance on personal projects and taste, to be sublimated into a shared vision. A “bottom up” approach, with “nice and consistent user interfaces” bolted on top of “all-time favorites”, with zero indication of how those nice and consistent UIs would work on extant code bases, all driven by somebody’s with a vision—drug induced or otherwise—that decides to lead the project towards its implementation.

It’s been nearly 30 years, but GNOME still works that way.

Sure, we’ve had a HIG for 25 years, and the shared development resources that the project provides tend to mask this, to the point that everyone outside the project assumes that all people with access to the GNOME commit bit work on the whole project, as a single unit. If you are here, listening (or reading) to this, you know it’s not true. In fact, it is so comically removed from the lived experience of everyone involved in the project that we generally joke about it.

Herding cats and vectors sum

During my first GUADEC, back in 2005, I saw a great slide from Seth Nickell, one of the original GNOME designers. It showed GNOME contributors represented as a jumble of vectors going in all directions, cancelling each component out; and the occasional movement in the project was the result of somebody pulling/pushing harder in their direction.

Of course, this is not the exclusive province of GNOME: you could take most complex free and open source software projects and draw a similar diagram. I contend, though, that when it comes to GNOME this is not emergent behaviour but it’s baked into the project from its very inception: a loosey-goosey collection of cats, herded together by whoever shows up with “a vision”, but, also, a collection of loosely coupled projects. Over the years we tried to put a rest to the notion that GNOME is a box of LEGO, meant to be assembled together by distributors and users in the way they most like it; while our software stack has graduated from the “thrown together at the last minute” quality of its first decade, our community is still very much following that very same model; the only way it seems to work is because we have a few people maintaining a lot of components.

On maintainers

I am a software nerd, and one of the side effects of this terminal condition is that I like optimisation problems. Optimising software is inherently boring, though, so I end up trying to optimise processes and people. The fundamental truth of process optimisation, just like software, is to avoid unnecessary work—which, in some cases, means optimising away the people involved.

I am afraid I will have to be blunt, here, so I am going to ask for your forgiveness in advance.

Let’s say you are a maintainer inside a community of maintainers. Dealing with people is hard, and the lord forbid you talk to other people about what you’re doing, what they are doing, and what you can do together, so you only have a few options available.

The first one is: you carve out your niche. You start, or take over, a project, or an aspect of a project, and you try very hard to make yourself indispensable, so that everything ends up passing through you, and everyone has to defer to your taste, opinion, or edict.

Another option: API design is opinionated, and reflects the thoughts of the person behind it. By designing platform API, you try to replicate your toughts, taste, and opinions into the minds of the people using it, like the eggs of parasitic wasp; because if everybody thinks like you, then there won’t be conflicts, and you won’t have to deal with details, like “how to make this application work”, or “how to share functionality”; or, you know, having to develop a theory of mind for relating to other people.

Another option: you try to reimplement the entirety of a platform by yourself. You start a bunch of projects, which require starting a bunch of dependencies, which require refactoring a bunch of libraries, which ends up cascading into half of the stack. Of course, since you’re by yourself, you end up with a consistent approach to everything. Everything is as it ought to be: fast, lean, efficient, a reflection of your taste, commitment, and ethos. You made everyone else redundant, which means people depend on you, but also nobody is interested in helping you out, because you are now taken for granted, on the one hand, and nobody is able to get a word edgewise into what you made on the other.

I purposefully did not name names, even though we can all recognise somebody in these examples. For instance, I recognise myself. I have been all of these examples, at one point or another over the past 20 years.

Painting a target on your back

But if this is what it looks like from within a project, what it looks like from the outside is even worse.

Once you start dragging other people, you raise your visibility; people start learning your name, because you appear in the issue tracker, on Matrix/IRC, on Discourse and Planet GNOME. Youtubers and journalists start asking you questions about the project. Randos on web forums start associating you to everything GNOME does, or does not; to features, design, and bugs. You become responsible for every decision, whether you are or not, and this leads to being the embodiment of all evil the project does. You’ll get hate mail, you’ll be harrassed, your words will be used against you and the project for ever and ever.

Burnout and you

Of course, that ends up burning people out; it would be absurd if it didn’t. Even in the best case possible, you’ll end up burning out just by reaching empathy fatigue, because everyone has access to you, and everyone has their own problems and bugs and features and wouldn’t it be great to solve every problem in the world? This is similar to working for non profits as opposed to the typical corporate burnout: you get into a feedback loop where you don’t want to distance yourself from the work you do because the work you do gives meaning to yourself and to the people that use it; and yet working on it hurts you. It also empowers bad faith actors to hound you down to the ends of the earth, until you realise that turning sand into computers was a terrible mistake, and we should have torched the first personal computer down on sight.

Governance

We want to have structure, so that people know what to expect and how to navigate the decision making process inside the project; we also want to avoid having a sacrificial lamb that takes on all the problems in the world on their shoulders until we burn them down to a cinder and they have to leave. We’re 28 years too late to have a benevolent dictator, self-appointed or otherwise, and we don’t want to have a public consultation every time we want to deal with a systemic feature. What do we do?

Examples

What do other projects have to teach us about governance? We are not the only complex free software project in existence, and it would be an appaling measure of narcissism to believe that we’re special in any way, shape or form.

Python

We should all know what a Python PEP is, but if you are not familiar with the process I strongly recommend going through it. It’s well documented, and pretty much the de facto standard for any complex free and open source project that has achieved escape velocity from a centralised figure in charge of the whole decision making process. The real achievement of the Python community is that it adopted this policy long before their centralised figure called it quits. The interesting thing of the PEP process is that it is used to codify the governance of the project itself; the PEP template is a PEP; teams are defined through PEPs; target platforms are defined through PEPs; deprecations are defined through PEPs; all project-wide processes are defined through PEPs.

Rust

Rust has a similar process for language, tooling, and standard library changes, called “RFC”. The RFC process is more lightweight on the formalities than Python’s PEPs, but it’s still very well defined. Rust, being a project that came into existence in a Post-PEP world, adopted the same type of process, and used it to codify teams, governance, and any and all project-wide processes.

Fedora

Fedora change proposals exist to discuss and document both self-contained changes (usually fairly uncontroversial, given that they are proposed by the same owners of module being changed) and system-wide changes. The main difference between them is that most of the elements of a system-wide change proposal are required, wheres for self-contained proposals they can be optional; for instance, a system-wide change must have a contingency plan, a way to test it, and the impact on documentation and release notes, whereas as self-contained change does not.

GNOME

Turns out that we once did have “GNOME Enhancement Proposals” (GEP), mainly modelled on Python’s PEP from 2002. If this comes as a surprise, that’s because they lasted for about a year, mainly because it was a reactionary process to try and funnel some of the large controversies of the 2.0 development cycle into a productive outlet that didn’t involve flames and people dramatically quitting the project. GEPs failed once the community fractured, and people started working in silos, either under their own direction or, more likely, under their management’s direction. What’s the point of discussing a project-wide change, when that change was going to be implemented by people already working together?

The GEP process mutated into the lightweight “module proposal” process, where people discussed adding and removing dependencies on the desktop development mailing list—something we also lost over the 2.x cycle, mainly because the amount of discussions over time tended towards zero. The people involved with the change knew what those modules brought to the release, and people unfamiliar with them were either giving out unsolicited advice, or were simply not reached by the desktop development mailing list. The discussions turned into external dependencies notifications, which also died up because apparently asking to compose an email to notify the release team that a new dependency was needed to build a core module was far too much of a bother for project maintainers.

The creation and failure of GEP and module proposals is both an indication of the need for structure inside GNOME, and how this need collides with the expectation that project maintainers have not just complete control over every aspect of their domain, but that they can also drag out the process until all the energy behind it has dissipated. Being in charge for the long run allows people to just run out the clock on everybody else.

Goals

So, what should be the goal of a proper technical governance model for the GNOME project?

Diffusing responsibilities

This should be goal zero of any attempt at structuring the technical governance of GNOME. We have too few people in too many critical positions. We can call it “efficiency”, we can call it “bus factor”, we can call it “bottleneck”, but the result is the same: the responsibility for anything is too concentrated. This is how you get conflict. This is how you get burnout. This is how you paralise a whole project. By having too few people in positions of responsibility, we don’t have enough slack in the governance model; it’s an illusion of efficiency.

Responsibility is not something to hoard: it’s something to distribute.

Empowering the community

The community of contributors should be able to know when and how a decision is made; it should be able to know what to do once a decision is made. Right now, the process is opaque because it’s done inside a million different rooms, and, more importantly, it is not recorded for posterity. Random GitLab issues should not be the only place where people can be informed that some decision was taken.

Empowering individuals

Individuals should be able to contribute to a decision without necessarily becoming responsible for a whole project. It’s daunting, and requires a measure of hubris that cannot be allowed to exist in a shared space. In a similar fashion, we should empower people that want to contribute to the project by reducing the amount of fluff coming from people with zero stakes in it, and are interested only in giving out an opinion on their perfectly spherical, frictionless desktop environment.

It is free and open source software, not free and open mic night down at the pub.

Actual decision making process

We say we work by rough consensus, but if a single person is responsible for multiple modules inside the project, we’re just deceiving ourselves. I should not be able to design something on my own, commit it to all projects I maintain, and then go home, regardless of whether what I designed is good or necessary.

Proposed GNOME Changes✝

✝ Name subject to change

PGCs

We have better tools than what the GEP used to use and be. We have better communication venues in 2025; we have better validation; we have better publishing mechanisms.

We can take a lightweight approach, with a well-defined process, and use it not for actual design or decision-making, but for discussion and documentation. If you are trying to design something and you use this process, you are by definition Doing It Wrong™. You should have a design ready, and series of steps to achieve it, as part of a proposal. You should already know the projects involved, and already have an idea of the effort needed to make something happen.

Once you have a formal proposal, you present it to the various stakeholders, and iterate over it to improve it, clarify it, and amend it, until you have something that has a rough consensus among all the parties involved. Once that’s done, the proposal is now in effect, and people can refer to it during the implementation, and in the future. This way, we don’t have to ask people to remember a decision made six months, two years, ten years ago: it’s already available.

Editorial team

Proposals need to be valid, in order to be presented to the community at large; that validation comes from an editorial team. The editors of the proposals are not there to evaluate its contents: they are there to ensure that the proposal is going through the expected steps, and that discussions related to it remain relevant and constrained within the accepted period and scope. They are there to steer the discussion, and avoid architecture astronauts parachuting into the issue tracker or Discourse to give their unwarranted opinion.

Once the proposal is open, the editorial team is responsible for its inclusion in the public website, and for keeping track of its state.

Steering group

The steering group is the final arbiter of a proposal. They are responsible for accepting it, or rejecting it, depending on the feedback from the various stakeholders. The steering group does not design or direct GNOME as a whole: they are the ones that ensure that communication between the parts happens in a meaningful manner, and that rough consensus is achieved.

The steering group is also, by design, not the release team: it is made of representatives from all the teams related to technical matters.

Is this enough?

Sadly, no.

Reviving a process for proposing changes in GNOME without addressing the shortcomings of its first iteration would inevitably lead to a repeat of its results.

We have better tooling, but the problem is still that we’re demanding that each project maintainer gets on board with a process that has no mechanism to enforce compliance.

Once again, the problem is that we have a bunch of fiefdoms that need to be opened up to ensure that more people can work on them.

Whither maintainers

In what was, in retrospect, possibly one of my least gracious and yet most prophetic moments on the desktop development mailing list, I once said that, if it were possible, I would have already replaced all GNOME maintainers with a shell script. Turns out that we did replace a lot of what maintainers used to do, and we used a large Python service to do that.

Individual maintainers should not exist in a complex project—for both the project’s and the contributors’ sake. They are inefficiency made manifest, a bottleneck, a point of contention in a distributed environment like GNOME. Luckily for us, we almost made them entirely redundant already! Thanks to the release service and CI pipelines, we don’t need a person spinning up a release archive and uploading it into a file server. We just need somebody to tag the source code repository, and anybody with the right permissions could do that.

We need people to review contributions; we need people to write release notes; we need people to triage the issue tracker; we need people to contribute features and bug fixes. None of those tasks require the “maintainer” role.

So, let’s get rid of maintainers once and for all. We can delegate the actual release tagging of core projects and applications to the GNOME release team; they are already releasing GNOME anyway, so what’s the point in having them wait every time for somebody else to do individual releases? All people need to do is to write down what changed in a release, and that should be part of a change itself; we have centralised release notes, and we can easily extract the list of bug fixes from the commit log. If you can ensure that a commit message is correct, you can also get in the habit of updating the NEWS file as part of a merge request.

Additional benefits of having all core releases done by a central authority are that we get people to update the release notes every time something changes; and that we can sign all releases with a GNOME key that downstreams can rely on.

Embracing special interest groups

But it’s still not enough.

Especially when it comes to the application development platform, we have already a bunch of components with an informal scheme of shared responsibility. Why not make that scheme official?

Let’s create the SDK special interest group; take all the developers for the base libraries that are part of GNOME—GLib, Pango, GTK, libadwaita—and formalise the group of people that currently does things like development, review, bug fixing, and documentation writing. Everyone in the group should feel empowered to work on all the projects that belong to that group. We already are, except we end up deferring to somebody that is usually too busy to cover every single module.

Other special interest groups should be formed around the desktop, the core applications, the development tools, the OS integration, the accessibility stack, the local search engine, the system settings.

Adding more people to these groups is not going to be complicated, or introduce instability, because the responsibility is now shared; we would not be taking somebody that is already overworked, or even potentially new to the community, and plopping them into the hot seat, ready for a burnout.

Each special interest group would have a representative in the steering group, alongside teams like documentation, design, and localisation, thus ensuring that each aspect of the project technical direction is included in any discussion. Each special interest group could also have additional sub-groups, like a web services group in the system settings group; or a networking group in the OS integration group.

What happens if I say no?

I get it. You like being in charge. You want to be the one calling the shots. You feel responsible for your project, and you don’t want other people to tell you what to do.

If this is how you feel, then there’s nothing wrong with parting ways with the GNOME project.

GNOME depends on a ton of projects hosted outside GNOME’s own infrastructure, and we communicate with people maintaining those projects every day. It’s 2025, not 1997: there’s no shortage of code hosting services in the world, we don’t need to have them all on GNOME infrastructure.

If you want to play with the other children, if you want to be part of GNOME, you get to play with a shared set of rules; and that means sharing all the toys, and not hoarding them for yourself.

Civil service

What we really want GNOME to be is a group of people working together. We already are, somewhat, but we can be better at it. We don’t want rule and design by committee, but we do need structure, and we need that structure to be based on expertise; to have distinct sphere of competence; to have continuity across time; and to be based on rules. We need something flexible, to take into account the needs of GNOME as a project, and be capable of growing in complexity so that nobody can be singled out, brigaded on, or burnt to a cinder on the sacrificial altar.

Our days of passing out in the middle of the dance floor are long gone. We might not all be old—actually, I’m fairly sure we aren’t—but GNOME has long ceased to be something we can throw together at the last minute just because somebody assumed the mantle of a protean ruler, and managed to involve themselves with every single project until they are the literal embodiement of an autocratic force capable of dragging everybody else towards a goal, until the burn out and have to leave for their own sake.

We can do better than this. We must do better.

To sum up

Stop releasing individual projects, and let the release team do it when needed.

Create teams to manage areas of interest, instead of single projects.

Create a steering group from representatives of those teams.

Every change that affects one or more teams has to be discussed and documented in a public setting among contributors, and then published for future reference.

None of this should be controversial because, outside of the publishing bit, it’s how we are already doing things. This proposal aims at making it official so that people can actually rely on it, instead of having to divine the process out of thin air.

The next steps

We’re close to the GNOME 49 release, now that GUADEC 2025 has ended, so people are busy working on tagging releases, fixing bugs, and the work on the release notes has started. Nevertheless, we can already start planning for an implementation of a new governance model for GNOME for the next cycle.

First of all, we need to create teams and special interest groups. We don’t have a formal process for that, so this is also a great chance at introducing the change proposal process as a mechanism for structuring the community, just like the Python and Rust communities do. Teams will need their own space for discussing issues, and share the load. The first team I’d like to start is an “introspection and language bindings” group, for all bindings hosted on GNOME infrastructure; it would act as a point of reference for all decisions involving projects that consume the GNOME software development platform through its machine-readable ABI description. Another group I’d like to create is an editorial group for the developer and user documentation; documentation benefits from a consistent editorial voice, while the process of writing documentation should be open to everybody in the community.

A very real issue that was raised during GUADEC is bootstrapping the steering committee; who gets to be on it, what is the committee’s remit, how it works. There are options, but if we want the steering committee to be a representation of the technical expertise of the GNOME community, it also has to be established by the very same community; in this sense, the board of directors, as representatives of the community, could work on defining the powers and compositions of this committee.

There are many more issues we are going to face, but I think we can start from these and evaluate our own version of a technical governance model that works for GNOME, and that can grow with the project. In the next couple of weeks I’ll start publishing drafts for team governance and the power/composition/procedure of the steering committee, mainly for iteration and comments.

Introspection’s edge

2023-10-25T09:31:00+01:00

tl;dr The introspection data for GLib (and its sub-libraries) is now generated by GLib itself instead of gobject-introspection; and in the near future, libgirepository is going to be part of GLib as well. Yes, there’s a circular dependency, but there are plans for dealing with that. Developers of language bindings should reach out to the GLib maintainers for future planning.

GLib-the-project is made of the following components:

glib, the base data types and C portability library
gmodule, a portable loadable module API
gobject, the object oriented run time type system
gio, a grab bag of I/O, IPC, file, network, settings, etc

Just like GLib, the gobject-introspection project is actually three components in a trench coat:

g-ir-scanner, the Python and flex based parser that generates the XML description of a GObject-based ABI
g-ir-compiler, the “compiler” that turns XML into an mmap()-able binary format
libgirepository, the C API for loading the binary format, resolving types and symbols, and calling into the underlying C library

Since gobject-introspection depends on GLib for its implementation, the introspection data for GLib has been, until now, generated during the gobject-introspection build. This has proven to be extremely messy:

as the introspection data is generated from the source code, gobject-introspection needs to have access to the GLib headers and source files
this means building against an installed copy of GLib for the declarations, and the GLib sources for the docblocks
to avoid having access to the sources at all times, there is a small script that extracts all the docblocks from a GLib checkout; this script has to be run manually, and its results committed to the gobject-introspection repository

This means that the gobject-introspection maintainers have to manually keep the GLib data in sync; that any change done to GLib’s annotations and documentation are updated typically at the end of the development cycle, which also means that any issue with the annotations is discovered almost always too late; and that any and all updates to the GLib introspection data require a gobject-introspection release to end up inside downstream distributions.

The gobject-introspection build can use GLib as a subproject, at the cost of some awful, awful build system messiness, but that does not help anybody working on GLib; and, of course, since gobject-introspection depends on GLib, we cannot build the former as a subproject of the latter.

As part of the push towards moving GLib’s API references towards the use of gi-docgen, GLib now checks for the availability of g-ir-scanner and g-ir-compiler on the installed system, and if they are present, they are used to generate the GLib, GObject, GModule, and GIO introspection data.

A good side benefit of having the actual project providing the ABI be in charge of generating its machine-readable description is that we can now decouple the release cycles of GLib and gobject-introspection; updates to the introspection data of GLib will be released alongside GLib, while the version of gobject-introspection can be updated whenever there’s new API inside libgirepository, instead of requiring a bump at every cycle to match GLib.

On the other hand, though, this change introduces a circular dependency between GLib and gobject-introspection; this is less than ideal, and though circular dependencies are a fact of life for whoever builds and packages complex software, it’d be better to avoid them; this means two options:

merging gobject-introspection into GLib
making gobject-introspection not depend on GLib

Sadly, the first option isn’t really workable, as it would mean making GLib depend on flex, bison, and CPython in order to build g-ir-scanner, or (worse) rewriting g-ir-scanner in C. There’s also the option of rewriting the C parser in pure Python, but that, too, isn’t exactly strife-free.

The second option is more convoluted, but it has the advantage of being realistically implemented:

move libgirepository into GLib as a new sub-library
move g-ir-compiler into GLib’s tools
copy-paste the basic GLib data types used by the introspection scanner into gobject-introspection

Aside from breaking the circular dependency, this allows language bindings that depend on libgirepository’s API (Python, JavaScript, Perl, etc) to limit their dependency to GLib, since they don’t need to deal with the introspection parser bits; it also affords us the ability to clean up the libgirepository API, provide better documentation, and formalise the file formats.

Speaking of file formats, and to manage expectations: I don’t plan to change the binary data layout; yes, we’re close to saturation when it comes to padding bits and fields, but there hasn’t been a real case in which this has been problematic. The API, on the other hand, really needs to be cleaned up, because it got badly namespaced when we originally thought it could be moved into GLib and then the effort was abandoned. This means that libgirepository will be bumped to 2.0 (matching the rest of GLib’s sub-libraries), and will require some work in the language bindings that consume it. For that, bindings developers should get in touch with GLib maintainers, so we can plan the migration appropriately.

The Mirror

2023-08-23T21:23:00+01:00

The GObject type system has been serving the GNOME community for more than 20 years. We have based an entire application development platform on top of the features it provides, and the rules that it enforces; we have integrated multiple programming languages on top of that, and in doing so, we expanded the scope of the GNOME platform in a myriad of directions. Unlike GTK, the GObject API hasn’t seen any major change since its introduction: aside from deprecations and little new functionality, the API is exactly the same today as it was when GLib 2.0 was released in March 2002. If you transported a GNOME developer from 2003 to 2023, they would have no problem understanding a newly written GObject class; though, they would likely appreciate the levels of boilerplate reduction, and the performance improvements that have been introduced over the years.

While having a stable API last this long is definitely a positive, it also imposes a burden on maintainers and users, because any change has to be weighted against the possibility of introducing unintended regressions in code that uses undefined, or undocumented, behaviour. There’s a lot of leeway when it comes to playing games with C, and GObject has dark corners everywhere.

The other burden is that any major change to a foundational library like GObject cascades across the entire platform. Releasing GLib 3.0 today would necessitate breaking API in the entirety of the GNOME stack and further beyond; it would require either a hard to execute “flag day”, or an impossibly long transition, reverberating across downstreams for years to come. Both solutions imply amounts of work that are simply not compatible with a volunteer-based project and ecosystem, especially the current one where volunteers of core components are now stretched thin across too many projects.

And yet, we are now at a cross-roads: our foundational code base has reached the point where recruiting new resources capable of affecting change on the project has become increasingly difficult; where any attempt at performance improvement is heavily counterbalanced by the high possibility of introducing world-breaking regressions; and where fixing the safety and ergonomics of idiomatic code requires unspooling twenty years of limitations inherent to the current design.

Something must be done if we want to improve the coding practices, the performance, and the safety of the platform without a complete rewrite.

The Mirror

‘Many things I can command the Mirror to reveal,’ she answered, ‘and to some I can show what they desire to see. But the Mirror also show things unbidden, and those are often stranger and more profitable than things we wish to behold. What you will see, if you leave the Mirror free to work, I cannot tell. For it shows things that were, and things that are, and things that yet may be. But which it is that he sees, even the wisest cannot always tell. Do you wish to look?’ — Lady Galadriel, “The Lords of the Rings”, Volume 1: The Fellowship of the Ring, Book 2: The Ring Goes South

In order to properly understand what we want to achieve, we need to understand the problem space that the type system is meant to solve, and the constraints upon which the type system was implemented. We do that by holding GObject up to Galadriel’s Mirror, and gazing into its surface.

Things that were

History became legend. Legend became myth. — Lady Galadriel, “The Lord of the Rings: The Fellowship of the Ring”

Before GObject there was GtkObject. It was a simpler time, it was a simpler stack. You added types only for the widgets and objects that related to the UI toolkit, and everything else was C89, with a touch of undefined behaviour, like calling function pointers with any number of arguments. Properties were “arguments”, likes were florps, and the timeline went sideways.

We had a class initialisation and an instance initialisation functions; properties were stored in a global hash table, but the property multiplexer pair of functions was stored on the type data instead of using the class structure. Types did not have private data: you only had keyed fields. No interfaces, only single inheritance. GtkObject was reference counted, and had an initially “floating” reference, to allow transparent ownership transfer from child to parent container when writing C code, and make the life of every language binding maintainer miserable in the process. There were weak references attached to an instance that worked by invoking a callback when the instance’s reference count reached zero. Signals operated exactly as they do today: large hash table of signal information, indexed by an integer.

None of this was thread safe. After all, GTK was not thread safe either, because X11 was not thread safe; and we’re talking about 1997: who even had hardware capable of multi-threading at the time? NPTL wasn’t even a thing, yet.

The introduction of GObject in 2001 changed some of the rules—mainly, around the idea of having dynamic types that could be loaded and unloaded in order to implement plugins. The basic design of the type system, after all, came from Beast, a plugin-heavy audio application, and it was extended to subsume the (mostly) static use cases of GTK. In order to support unloading, the class aspect of the type system was allowed to be cleaned up, but the type data had to be registered and never unloaded; in other words, once a type was registered, it was there forever.

“Arguments” were renamed to properties, and were extended to include more than basic types, provide validations, and notify of changes; the overall design was still using a global hash table to store all the properties across all types. Properties were tied to the GObject type, but the property definition existed as a separate type hierarchy that was designed to validate values, but not manage fields inside a class. Signals were ported wholesale, with minimal changes mainly around the marshalling of values and abstracting closures.

The entire plan was to have GObject as one of the base classes at the root of a specific hierarchy, with all the required functionality for GTK to inherit from for its own GtkObject, while leaving open the possibility of creating other hierarchies, or even other roots with different functionality, for more lightweight objects.

These constraints were entirely intentional; the idea was to be able to port GTK to the new type system, and to an out of tree GLib, during the 1.3 development phase, and minimise the amount of changes necessary to make the transition work not just inside GTK, but inside of GNOME too.

Little by little, the entire GObject layer was ported towards thread safety in the only way that worked without breaking the type system: add global locks around everything; use read-write locks for the type data; lock the access and traversal of the property hash table and of the signals table. The only real world code bases that actively exercised multi-threading support were GStreamer and the GNOME VFS API that was mainly used by Nautilus.

With the 3.0 API, GTK dropped the GtkObject base type: the whole floating reference mechanism was moved to GObject, and a new type was introduced to provide the “initial” floating reference to derived types. Around the same time, a thread-safe version of weak references for GObject appeared as a separate API, which confused the matter even more.

Things that are

Darkness crept back into the forests of the world. Rumour grew of a Shadow in the East, whispers of a Nameless Fear. — Lady Galadriel, “The Lord of the Rings: The Fellowship of the Ring”

Let’s address the elephant in the room: it’s completely immaterial how many lines of code you have to deal with when creating a new type. It’s a one-off cost, and for most cases, it’s a matter of using the existing macros. The declaration and definition macros have the advantages of enforcing a series of best practices, and keep the code consistent across multiple projects. If you don’t want to deal with boilerplate when using C, you chose the wrong language to begin with. The existence of excessive API is mainly a requirement to allow other languages to integrate their type system with GObject’s own.

The dynamic part of the type system has gotten progressively less relevant. Yes: you can still create plugins, and those can register types; but classes are never unloaded, just like their type data. There is some attempt at enforcing an order of operations: you cannot just add an interface after a type has been instantiated any more; and while you can add properties and signals after class initialisation, it’s mainly a functionality reserved for specific language bindings to maintain backward compatibility.

Yes, defining properties is boring, and could probably be simplified, but the real cost is not in defining and installing a GParamSpec: it’s in writing the set and get property multiplexers, validating the values, boxing and unboxing the data, and dealing with the different property flags; none of those things can be wrapped in some fancy C preprocessor macros—unless you go into the weeds with X macros. The other, big cost of properties is their storage inside a separate, global, lock-happy hash table. The main use case of this functionality—adding entirely separate classes of properties with the same semantics as GObject properties, like style properties and child properties in GTK—has completely fallen out of favour, and for good reasons: it cannot be managed by generic code; it cannot be handled by documentation generators without prior knowledge; and, for the same reason, it cannot be introspected. Even calling these “properties” is kind of a misnomer: they are value validation objects that operate only when using the generic (and less performant) GObject accessor API, something that is constrained to things like UI definition files in GTK, or language bindings. If you use the C accessors for your own GObject type, you’ll have to implement validation yourself; and since idiomatic code will have the generic GObject accessors call the public C API of your type, you get twice the amount of validation for no reason whatsoever.

Signals have mostly been left alone, outside of performance improvements that were hard to achieve within the constraints of the existing implementation; the generic FFI-based closure turned out to be a net performance loss, and we’re trying to walk it back even for D-Bus, which was the main driver for it to land in the first place. Marshallers are now generated with a variadic arguments variant, to reduce the amount of boxing and unboxing of GValue containers. Still, there’s not much left to squeeze out of the old GSignal API.

The atomic nature of the reference counting can be a costly feature, especially for code bases that are by necessity single-threaded; the fact that the reference count field is part of the (somewhat) public API prevents fundamental refactorings, like switching to biased reference counting for faster operations on the same thread that created an instance. The lack of room on GObject also prevents storing the thread ID that owns the instance, which in turn prevents calling the GObjectClass.dispose() and GObjectClass.finalize() virtual functions on the right thread, and requires scheduling the destruction of an object on a separate main context, or locking the contents of an object at a further cost.

Things that yet may be

The quest stands upon the edge of a knife: stray but a little, and it will fail to the ruin of all. Yet hope remains, while the company is true. — Lady Galadriel, “The Lord of the Rings: The Fellowship of the Ring”

Over the years, we have been strictly focusing on GObject: speeding up its internals, figuring out ways to improve property registration and performance, adding new API and features to ensure it behaved more reliably. The type system has also been improved, mainly to streamline its use in idiomatic GObject code bases. Not everything worked: properties are still a problem; weak references and pointers are a mess, with two different API that interact badly with GObject; signals still exists on a completely separate plane; GObject is still wildly inefficient when it comes to locking.

The thesis of this strawman is that we reached the limits of backwards compatibility of GObject, and any attempt at improving it will inevitably lead to a more brittle code, rife with potential regressions. The typical answer, in this case, would be to bump the API/ABI of GObject, remove the mistakes of the past, and provide a new idiomatic approach. Sadly, doing so not only would require a level of resources we, as the GLib project stewards, cannot provide, but it would also completely break the entire ecosystem in a way that is not recoverable. Either nobody would port to the new GObject-3.0 API; or the various projects that depend on GObject would inevitably fracture, following whichever API version they can commit to; in the meantime, downstream distributors would suffer the worst effects of the shared platform we call “Linux”.

Between inaction and slow death, and action with catastrophic consequences, there’s the possibility of a third option: what if we stopped trying to emulate Java, and have a single “god” type?

Our type system is flexible enough to support partitioning various responsibilities, and we can defer complexity where it belongs: into faster moving dependencies, that have the benefit of being able to iterate and change at a much higher rate than the foundational library of the platform. What’s the point of shoving every possible feature into the base class, in order to cover ever increasingly complex use cases across multiple languages, when we can let consumers decide to opt into their own well-defined behaviours? What GObject ought to provide is a set of reliable types that can be combined in expressive ways, and that can be inspected by generic API.

A new, old base type

We already have a derivable type, called GTypeInstance. Typed instances don’t have any memory management: once instantiated, they can only be moved, or freed. All our objects already are typed instances, since GObject inherits from it. Contrary to the current common practices we should move towards using GTypeInstance for our types.

There’s a distinct lack of convenience API for defining typed instances, mostly derived from the fact that GTypeInstance is seen as a form of “escape hatch” for projects to use in order to avoid GObject. In practice, there’s nothing that prevents us from improving the convenience of creating new instantiatable/derivable types, especially if we start using them more often. The verbose API must still exist, to allow language bindings and introspection to handle this kind of types, but just like we made convenience macros for declaring and defining GObject types, we can provide macros for new typed instances, and for setting up a GValue table.

Optional functionality

Typed instances require a wrapper API to free their contents before calling g_type_free_instance(). Nothing prevents us from adding a GFinalizable interface that can be implemented by a GTypeInstance, though: interfaces exist at the type system level, and do not require GObject to work.

typedef struct {
  void (* finalize) (GFinalizable *self);
} GFinalizableInterface;

If a typed instance provides an implementation of GFinalizable, then g_type_free_instance() can free the contents of the instance by calling g_finalizable_finalize().

This interface is optional, in case your typed instance just contains simple values, like:

typedef struct {
  GTypeInstance parent;

  bool is_valid;
  double x1, y1;
  double x2, y2;
} Box;

and does not require deallocations outside of the instance block.

A similar interface can be introduced for cloning instances, allowing a copy operation alongside a move:

typedef struct {
  GClonable * (* clone) (GClonable *self);
} GClonable;

We could then introduce g_type_instance_clone() as a generic entry point that either used GClonable, or simply allocated a new instance and called memcpy() on it, using the size of the instance (and eventual private data) known to the type system.

The prior art for this kind of functionality exists in GIO, in the form of the GInitable and GAsyncInitable interfaces; unfortunately, those interfaces require GObject, and they depend on GCancellable and GAsyncResult objects, which prevent us from moving them into the lower level API.

Typed containers and life time management

The main functionality provided by GObject is garbage collection through reference counting: you acquire a (strong) reference when you need to access an instance, and release it when you don’t need the instance any more. If the reference you released was the last one, the instance gets finalized.

Of course, once you introduce strong references you open the door to a veritable bestiary of other type of references:

weak references, used to keep a “pointer” to the instance, and get a notification when the last reference drops
floating references, used as a C convenience to allow ownership transfer of newly constructed “child” objects to their “parent”
toggle references, used by language bindings that acquire a strong reference on an instance they wrap with a native object; when the toggle reference gets triggered it means that the last reference being held is the one on the native wrapper, and the wrapper can be dropped causing the instance to be finalized

All of these types of reference exist inside GObject, but since they were introduced over the years, they are bolted on top of the base class using the keyed data storage, which comes with its own costly locking and ordering; they are also managed through the finalisation code, which means there are re-entrancy issues or undefined ordering behaviours that routinely crop up over the years, especially when trying to optimise construction and destruction phases.

None of this complexity is, strictly speaking, necessary; we don’t care about an instance being reference counted: a “parent” object can move the memory of a “child” typed instance directly into its own code. What we care about is that, whenever other code interacts with ours, we can hand out a reference to that memory, so that ownership is maintained.

Other languages and standard libraries have the same concept:

C++’s shared_ptr
Rust’s std::rc::Rc

These constructs are not part of a base class: they are wrappers around instances. This means you’re not handing out a reference to an instance: you are handing out a reference to a container, which holds the instance for you. The behaviour of the value is made explicit by the type system, not implicit to the type.

A simple implementation of a typed “reference counted” container would provide us with both strong and weak references:

typedef struct _GRc GRc;
typedef struct _GWeak GWeak;

GRc *g_rc_new (GType data_type, gpointer data);

GRc *g_rc_acquire (GRc *rc);
void g_rc_release (GRc *rc);

gpointer g_rc_get_data (GRc *rc);

GWeak *g_rc_downgrade (GRc *rc);
GRc *g_weak_upgrade (GWeak *weak);

bool g_weak_is_empty (GWeak *weak);
gpointer g_weak_get_data (GWeak *weak);

Alongside this type of containers, we could also have a specialisation for atomic reference counted containers; or pinned containers, which guarantee that an object is kept in the same memory location; or re-implement referenced containers inside each language binding, to ensure that the behaviour is tailored to the memory management of those languages.

Specialised types

Container types introduce the requirement of having the type system understand that an object can be the product of two types: the type of the container, and the type of the data. In order to allow properties, signals, and values to effectively provide introspection of this kind of container types we are going to need to introduce “specialised” types:

GRc exists as a “generic”, abstract type in the type system
any instance of GRc that contains a instance of type A gets a new type in the type system

A basic implementation would look like:

GRc *
g_rc_new (GType data_type, gpointer data)
{
  // Returns an existing GType if something else already
  // has registered the same GRc<T>
  GType rc_type =
    g_generic_type_register_static (G_TYPE_RC, data_type);

  // Instantiates GRc, but gives it the type of
  // GRc<T>; there is only the base GRc class
  // and instance initialization functions, as
  // GRc<T> is not a pure derived type
  GRc *res = (GRc *) g_type_create_instance (rc_type);
  res->data = data;

  return res;
}

Any instance of type GRc<A> satisfies the “is-a” relationship with GRc, but it is not a purely derived type:

GType rc_type =
  ((GTypeInstance *) rc)->g_class.g_type;
g_assert_true (g_type_is_a (rc_type, G_TYPE_RC));

The GRc<A> type does not have a different instance or class size, or its own class and instance initialisation functions; it’s still an instance of the GRc type, with a different GType. The GRc<A> type only exists at run time, as it is the result of the type instantiation; you cannot instantiate a plain GRc, or derive your type from GRc in order to create your own reference counted type, either:

// WRONG
GRc *rc = g_type_create_instance (G_TYPE_RC);

// WRONG
typedef GRc GtkWidget;

You can only use a GRc inside your own instance:

typedef struct {
  // GRc<GtkWidget>
  GRc *parent;
  // GRc<GtkWidget>
  GRc *first_child;
  // GRc<GtkWidget>
  GRc *next_sibling;

  // ...
} GtkWidgetPrivate;

Tuple types

Tuples are generic containers of N values, but right now we don’t have any way of formally declaring them into the type system. A hack is to use arrays of similarly typed values, but with the deprecation of GValueArray—which is a bad type that does not allow reference counting, and does not give you guarantees anyway—we only have C arrays and pointer types.

Registering a new tuple type would work like a generic type: a base GTuple abstract type as the “parent”, and a number of types:

typedef struct _GTuple GTuple;

GTuple *
g_tuple_new_int (size_t n_elements,
                 int elements[])
{
  GType tuple_type =
    g_tuple_type_register_static (G_TYPE_TUPLE, n_elements, G_TYPE_INT);

  GTuple *res = g_type_create_instance (tuple_type);
  for (size_t i = 0; i < n_elements; i++)
    g_tuple_add (res, elements[i]);

  return res;
}

We can also create specialised tuple types, like pairs:

typedef struct _GPair GPair;

GPair *
g_pair_new (GType this_type,
            GType that_type,
            ...);

This would give use the ability to standardise our API around fundamental types, and reduce the amount of ad hoc container types that libraries have to define and bindings have to wrap with native constructs.

Sum types

Of course, once we start with specialised types, we end up with sum types:

typedef enum {
  SQUARE,
  RECT,
  CIRCLE,
} ShapeKind;

typedef struct {
  GTypeInstance parent;

  ShapeKind kind;

  union {
    struct { Point origin; float side; };
    struct { Point origin; Size size; };
    struct { Point center; float radius; };
  } shape;
} Shape;

As of right now, discriminated unions don’t have any special handling in the type system: they are generally boxed types, or typed instances, but they require type-specific API to deal with the discriminator field and type. Since we have types for enumerations and instances, we can register them at the same time, and provide offsets for direct access:

GType
g_sum_type_register_static (const char *name,
                            size_t class_size,
                            size_t instance_size,
                            GType tag_enum_type,
                            offset_t tag_field);

This way it’s possible to ask the type system for:

the offset of the tag in an instance, for direct access
all the possible values of the tag, by inspecting its GEnum type

From then on, we can easily build types like Option and Result:

typedef enum {
  G_RESULT_OK,
  G_RESULT_ERR
} GResultKind;

typedef struct {
  GTypeInstance parent;

  GResultKind type;
  union {
    GValue value;
    GError *error;
  } result;
} GResult;

// ...
g_sum_type_register_static ("GResult",
                            sizeof (GResultClass),
                            sizeof (GResult),
                            G_TYPE_RESULT_KIND,
                            offsetof (GResult, type));

// ...
GResult *
g_result_new_boolean (gboolean value)
{
  GType res_type =
    g_generic_type_register_static (G_TYPE_RESULT,
                                    G_TYPE_BOOLEAN)
  GResult *res =
    g_type_create_instance (res_type);
  g_value_set_boolean (&res->result.value, value);

  return res;
}

// ...
g_autoptr (GResult) result = obj_finish (task);
switch (g_result_get_kind (result)) {
  case G_RESULT_OK:
    g_print ("Result: %s\n",
      g_result_get_boolean (result)
        ? "true"
        : "false");
    break;

  case G_RESULT_ERROR:
    g_printerr ("Error: %s\n",
      g_result_get_error_message (result));
    break;
}

// ...
g_autoptr (GResult) result =
  g_input_stream_read_bytes (stream);
if (g_result_is_error (result)) {
  // ...
} else {
  g_autoptr (GBytes) data = g_result_get_boxed (result);
  // ...
}

Consolidating GLib and GType

Having the type system in a separate shared library did make sense back when GLib was spun off from GTK; after all, GLib was mainly a set of convenient data types for a language that lacked a decent standard library. Additionally, not many C projects were interested in the type system, as it was perceived as a big chunk of functionality in an era where space was at a premium. These days, the smallest environment capable of running GLib code is plenty capable of running the GObject type system as well. The separation between GLib data types and the GObject type system has created data types that are not type safe, and work by copying data, by having run time defined destructor functions, or by storing pointers and assuming everything will be fine. This leads to code duplication between shared libraries, and prevents the use of GLib data types in the public API, lest the introspection information gets lost.

Moving the type system inside GLib would allow us to have properly typed generic container types, like a GVector replacing GArray, GPtrArray, GByteArray, as well as the deprecated GValueArray; or a GMap and a GSet, replacing GHashTable, GSequence, and GtkRBTree. Even the various list models could be assembled on top of these new types, and moved out of GTK.

Current consumers of GLib-only API would still have their basic C types, but if they don’t want to link against a slightly bigger shared library that includes GTypeInstance, GTypeInterface, and the newly added generic, tuple, and sum types, then they would probably be better served by projects like c-util instead.

Properties

Instead of bolting properties on top of GParamSpec, we can move their definition into the type system; after all, properties are a fundamental part of a type, so it does not make sense to bind them to the class instantiation. This would also remove the long-standing issue of properties being available for registration long after a class has been initialised; it would give us the chance to ship a utility for inspecting the type system to get all the meta-information on the hierarchy and generating introspection XML without having to compile a small binary.

If we move property registration to the type registration we can also finally move away from multiplexed accessors, and use direct instance field access where applicable:

GPropertyBuilder builder;

g_property_builder_init (&builder,
  G_TYPE_STRING, "name");
// Stop using flags, and use proper setters; since
// there's no use case for unsetting the readability
// flag, we don't even need a boolean argument
g_property_builder_set_readwrite (&builder);
// The offset is used for read and write access...
g_property_builder_set_private_offset (&builder,
  offsetof (GtkWidgetPrivate, name));
// ... unless an accessor function is provided; in
// this case we want setting a property to go through
// a function
g_property_builder_set_setter_func (&builder,
  gtk_widget_set_name);

// Register the property into the type; we return the
// offset of the property into the type node, so we can
// access the property definition with a fast look up
properties[NAME] =
  g_type_add_instance_property (type,
    g_property_builder_end (&builder));

Accessing the property information would then be a case of looking into the type system under a single reader lock, instead of traversing all properties in a glorified globally locked hash table.

Once we have a property registered in the type system, accessing it is a matter of calling API on the GProperty object:

void
gtk_widget_set_name (GtkWidget *widget,
                     const char *name)
{
  GProperty *prop =
    g_type_get_instance_property (GTK_TYPE_WIDGET,
                                  properties[NAME]);

  g_property_set (prop, name);
}

Signals

Moving signal registration into the type system would allow us to subsume the global locking into the type locks; it would also give us the chance to simplify some of the complexity for re-emission and hooks:

GSignalBuilder builder;

g_signal_builder_init (&builder, "insert-text");
g_signal_builder_set_args (&builder, 3,
  (GSignalArg[]) {
    { .name = "text", .gtype = G_TYPE_STRING },
    { .name = "length", .gtype = G_TYPE_SIZE },
    { .name = "position", .gtype = G_TYPE_OFFSET },
  });
g_signal_builder_set_retval (&builder,
  G_TYPE_OFFSET);
g_signal_builder_set_class_offset (&builder,
  offsetof (EditableClass, insert_text));

signals[INSERT_TEXT] =
  g_type_add_class_signal (type,
    g_signal_builder_end (&builder));

By taking the chance of moving signals out of the their own namespace we can also move to a model where each class is responsible for providing the API necessary to connect and emit signals, as well as providing callback types for each signal. This would allow us to increase type safety, and reduce the reliance on generic API:

typedef offset_t (* EditableInsertText) (Editable *self,
                                         const char *text,
                                         size_t length,
                                         offset_t position);

unsigned long
editable_connect_insert_text (Editable *self,
                              EditableInsertText callback,
                              gpointer user_data,
                              GSignalFlags flags);

offset_t
editable_emit_insert_text (Editable *self,
                           const char *text,
                           size_t length,
                           offset_t position);

Extending the type system

Some of the metadata necessary to provide properly typed properties and signals is missing from the type system. For instance, by design, there is no type representing a uint16_t; we are supposed to create a GParamSpec to validate the value of a G_TYPE_INT in order to fit in the 16bit range. Of course, this leads to excessive run time validation, and relies on C’s own promotion rules for variadic arguments; it also does not work for signals, as those do not use GParamSpec. More importantly, though, the missing connection between C types and GTypes prevents gathering proper introspection information for properties and signal arguments: if we only have the GType we cannot generate the full metadata that can be used by documentation and language bindings, unless we’re willing to lose specificity.

Not only the type system should be sufficient to contain all the standard C types that are now available, we also need the type system to provide us with enough information to be able to serialise those types into the introspection data, if we want to be able to generate code like signal API, type safe bindings, or accurate documentation for properties and signal handlers.

Introspection

Introspection exists outside of GObject mainly because of dependencies; the parser, abstract syntax tree, and transformers are written in Python and interface with a low level C tokeniser. Adding a CPython dependency to GObject is too much of a stretch, especially when it comes to bootstrapping a system. While we could keep the dependency optional, and allow building GObject without support for introspection, keeping the code separate is a simpler solution.

Nevertheless, GObject should not ignore introspection. The current reflection API inside GObject should generate data that is compatible with the libgirepository API and with its GIR parser. Currently, gobject-introspection is tasked with generating a small C executable, compiling it, running it to extract metadata from the type system, as well as the properties and signals of a GObject type, and generate XML that can be parsed and included into the larger GIR metadata for the rest of the ABI being introspected. GObject should ship a pre-built binary, instead; it should dlopen the given library or executable, extract all the type information, and emit the introspection data. This would not make gobject-introspection more cross-compilable, but it would simplify its internals and its distributability. We would not need to know how to compile and run C code from a Python script, for one; a simple executable wrapper around a native copy of the GObject-provided binary would be enough.

Ideally, we could move the girepository API into GObject itself, and allow it to load the binary data compiled out of the XML; language bindings loading the data at run time would then need to depend on GObject instead of an additional library, and we could ship the GIR → typelib compiler directly with GLib, leaving gobject-introspection to deal only with the parsing of C headers, docblocks, and annotations, to generate the XML representation of the C/GObject ABI.

There and back again

And the ship went out into the High Sea and passed on into the West, until at last on a night of rain Frodo smelled a sweet fragrance on the air and heard the sound of singing that came over the water. And then it seemed to him that as in his dream in the house of Bombadil, the grey rain-curtain turned all to silver glass and was rolled back, and he beheld white shores and beyond them a far green country under a swift sunrise. — “The Lord of the Rings”, Volume 3: The Return of the King, Book 6: The End of the Third Age

The hard part of changing a project in a backward compatible way is resisting the temptation of fixing the existing design. Some times it’s necessary to backtrack the chain of decisions, and consider the extant code base a dead branch; not because the code is wrong, or bug free, but because any attempt at doubling down on the same design will inevitably lead to breakage. In this sense, it’s easy to just declare “maintenance bankruptcy”, and start from a new major API version: breaks allow us to fix the implementation, at the cost of adapting to new API. For instance, widgets are still the core of GTK, even after 4 major revisions; we did not rename them to “elements” or “actors”, and we did not change how the windows are structured. You are still supposed to build a tree of widgets, connect callbacks to signals, and let the main event loop run. Porting has been painful because of underlying changes in the graphics stack, or because of portability concerns, but even with the direction change of favouring composition over inheritance, the knowledge on how to use GTK has been transferred from GTK 1 to 4.

We cannot do the same for GObject. Changing how it is implemented implies changing everything that depends on it; it means introducing behavioural changes in subtle, and hard to predict ways. Luckily for us, the underlying type system is still flexible and nimble enough that it can give us the ability to change direction, and implement an entirely different approach to object orientation—one that is more in line with languages like modern C++ and Rust. By following new approaches we can slowly migrate our platform to other languages over time, with a smaller impedance mismatch caused by the current design of our object model. Additionally, by keeping the root of the type system, we maintain the ability to provide a stable C ABI that can be consumed by multiple languages, which is the strong selling point of the GNOME ecosystem.

Why do all of this work, though? Compared to a full API break, this proposal has the advantage of being tractable and realistic; I cannot overemphasise enough how little appetite there is for a “GObject 3.0” in the ecosystem. The recent API bump from libsoup2 to libsoup3 has clearly identified that changes deep into the stack end up being too costly an effort: some projects have found it easier to switch to another HTTP library altogether, rather than support two versions of libsoup for a while; other projects have decided to drop compatibility with libsoup2, forcing the hand of every reverse dependency both upstream and downstream. Breaking GObject would end up breaking the ecosystem, with the hope of a “perfect” implementation way down the line and with very few users on one side, and a dead branch used by everybody else on the other.

Of course, the complexity of the change is not going to be trivial, and it will impact things like the introspection metadata and the various language bindings that exist today; some bindings may even require a complete redesign. Nevertheless, by implementing this new object model and leaving GObject alone, we buy ourselves enough time and space to port our software development platform towards a different future.

Maybe this way we will get to save the Shire; and even if we give up some things, or even lose them, we still get to keep what matters.

Configuring portals

2023-05-29T18:31:00+01:00

One of the things I’ve been recently working on at Igalia is the desktop portals implementation, the middleware layer of API for application and toolkit developers that allows sandboxed applications to interact with the host system. Sandboxing technologies like Flatpak and Snap expose the portal D-Bus interfaces inside the sandbox they manage, to handle user-mediated interactions like opening a file that exists outside of the locations available to the sandboxed process, or talking to privileged components like the compositor to obtain a screenshot.

Outside of allowing dynamic permissions for sandboxed applications, portals act as a vendor-neutral API for applications to target when dealing with Linux as an OS; this is mostly helpful for commercial applications that are not tied to a specific desktop environment, but don’t want to re-implement the layer of system integration from the first principles of POSIX primitives.

The architecture of desktop portals has been described pretty well in a blog post by Peter Hutterer, but to recap:

desktop portals are a series of D-Bus interfaces
toolkits and applications call methods on those D-Bus interfaces
there is a user session daemon called xdg-desktop-portal that provides a service for the D-Bus interfaces
xdg-desktop-portal implements some of those interface directly
for the interfaces that involve user interaction, or interaction with desktop-specific services, we have separate services that are proxied by xdg-desktop-portal; GNOME has xdg-desktop-portal-gnome, KDE has xdg-desktop-portal-kde; Sway and wlroot-based compositors have xdg-desktop-portal-wlr; and so on, and so forth

There’s also xdg-desktop-portal-gtk, which acts a bit as a reference portal implementation, and a shared desktop portal implementation for a lot of GTK-based environments. Ideally, every desktop environment should have their own desktop portal implementation, so that applications using the portal API can be fully integrated with each desktop’s interface guidelines and specialised services.

One thing that is currently messy is the mechanism by which xdg-desktop-portal finds the portal implementations available on the system, and decides which implementation should be used for a specific interface.

Up until the current stable version of xdg-desktop-portal, the configuration worked this way:

each portal implementation (xdg-desktop-portal-gtk, -gnome, -kde, …) ships a ${NAME}.portal file; the file is a simple INI-like desktop entry file with the following keys:
- DBusName, which contains the service name of the portal, for instance, org.freedesktop.impl.portal.desktop.gnome for the GNOME portals; this name is used by xdg-desktop-portal to launch the portal implementation
- Interfaces, which contains a list of D-Bus interfaces under the org.freedesktop.impl.portal.* namespace that are implemented by the desktop-specific portal; xdg-desktop-portal will match the portal implementation with the public facing D-Bus interface internally
- UseIn, which contains the name of the desktop to be matched with the contents of the $XDG_CURRENT_DESKTOP environment variable
once xdg-desktop-portal starts, it finds all the .portal files in a well-known location and builds a list of portal implementations currently installed in the system, containing all the interfaces they implement as well as their preferred desktop environment
whenever something calls a method on an interface in the org.freedesktop.portal.* namespace, xdg-desktop-portal will check the current desktop using the XDG_CURRENT_DESKTOP environment variable, and check if the portal that has a UseIn key that matches the current desktop
once there’s a match, xdg-desktop-portal will activate the portal implementation and proxy the calls made on the org.freedesktop.portal interfaces over to the org.freedesktop.impl.portal ones

This works perfectly fine for the average case of a Linux installation with a single session, using a single desktop environment, and a single desktop portal. Where things get messy is the case where you have multiple sessions on the same system, each with its own desktop and portals, or even no portals whatsoever. In a bad scenario, you may get the wrong desktop portal just because the name sorts before the one you’re interested in, so you get the GTK “reference” portals instead of the KDE-specific ones; in the worst case scenario, you may get a stall when launching an application just because the wrong desktop portal is trying to contact a session service that simply does not exist, and you have to wait 30 seconds for a D-Bus timeout.

The problem is that some desktop portal implementations are shared across desktops, or cover only a limited amount of interfaces; a mandatory list of desktop environments is far too coarse a tool to deal with this. Additionally, xdg-desktop-portal has to have enough fallbacks to ensure that, if it cannot find any implementation for the current desktop, it will proxy to the first implementation it can find in order to give a meaningful answer. Finally, since the supported desktops are shipped by the portal themselves, there’s no way to override this information by packagers, admins, or users.

After iterating over the issue, I ended up writing the support for a new configuration file. Instead of having portals say what kind of desktop environment they require, we have desktop environments saying which portal implementations they prefer. Now, each desktop should ship a ${NAME}-portals.conf INI-like desktop entry file listing each interface, and what kind of desktop portal should be used for it; for instance, the GNOME desktop should ship a gnome-portals.conf configuration file that specifies a default for every interface:

[preferred]
default=gnome

On the other hand, you could have a Foo desktop that relies on the GTK portal for everything, except for specific interfaces that are implemented by the “foo” portal:

[preferred]
default=gtk
org.freedesktop.impl.portal.Screenshot=foo
org.freedesktop.impl.portal.Screencast=foo

You could also disable all portals except for a specific interface (and its dependencies):

[preferred]
default=none
org.freedesktop.impl.portal.Account=gtk
org.freedesktop.impl.portal.FileChooser=gtk
org.freedesktop.impl.portal.Lockdown=gtk
org.freedesktop.impl.portal.Settings=gtk

Or, finally, you could disable all portal implementations:

[preferred]
default=none

A nice side effect of this work is that you can configure your own system, by dropping a portals.conf configuration file inside the XDG_CONFIG_HOME/xdg-desktop-portal directory; this should cover all the cases in which people assemble their desktop out of disparate components.

By having desktop environments (or, in a pinch, the user themselves) owning the kind of portals they require, we can avoid messy configurations in the portal implementations, and clarify the intended behaviour to downstream packagers; at the same time, generic portal implementations can be adopted by multiple environments without necessarily having to know which ones upfront.

In a way, the desktop portals project is trying to fulfill the original mission of freedesktop.org’s Cross-desktop Group: a set of API that are not bound to a single environment, and can be used to define “the Linux desktop” as a platform.

Of course, there’s a lot of work involved in creating a vendor-neutral platform API, especially when it comes to designing both the user and the developer experiences; ideally, more people should be involved in this effort, so if you want to contribute to the Linux ecosystem, this is an area where you can make the difference.

Writing Bindable API, 2023 Edition

2023-02-20T00:35:00+00:00

First of all, you should go on the gobject-introspection website and read the page on how to write bindable API. What I’m going to write here is going to build upon what’s already documented, or will update the best practices, so if you maintain a GObject/C library, or you’re writing one, you must be familiar with the basics of gobject-introspection. It’s 2023: it’s already too bad we’re still writing C libraries, we should at the very least be responsible about it.

A specific note for people maintaining an existing GObject/C library with an API designed before the mainstream establishment of gobject-introspection (basically, anything written prior to 2011): you should really consider writing all new types and entry points with gobject-introspection in mind, and you should also consider phasing out older API and replacing it piecemeal with a bindable one. You should have done this 10 years ago, and I can already hear the objections, but: too bad. Just because you made an effort 10 years ago it doesn’t mean things are frozen in time, and you don’t get to fix things. Maintenance means constantly tending to your code, and that doubly applies if you’re exposing an API to other people.

Let’s take the “how to write bindable API” recommendations, and elaborate them a bit.

Structures with custom memory management

The recommendation is to use GBoxed as a way to specify a copy and a free function, in order to clearly define the memory management semantics of a type.

The important caveat is that boxed types are necessary for:

opaque types that can only be heap allocated
using a type as a GObject property
using a type as an argument or return value for a GObject signal

You don’t need a boxed type for the following cases:

your type is an argument or return value for a method, function, or virtual function
your type can be placed on the stack, or can be allocated with malloc()/free()

Additionally, starting with gobject-introspection 1.76, you can specify the copy and free function of a type without necessarily registering a boxed type, which leaves boxed types for the thing they were created: signals and properties.

Addendum: object types

Boxed types should only ever be used for plain old data types; if you need inheritance, then the strong recommendation is to use GObject. You can use GTypeInstance, but only if you know what you’re doing; for more information on that, see my old blog post about typed instances.

Functionality only accessible through a C macro

This ought to be fairly uncontroversial. C pre-processor symbols don’t exist at the ABI level, and gobject-introspection is a mechanism to describe a C ABI. Never, ever expose API only through C macros; those are for C developers. C macros can be used to create convenience wrappers, but remember that anything they call must be public API, and that other people will need to re-implement the convenience wrappers themselves, so don’t overdo it. C developers deserve some convenience, but not at the expense of everyone else.

Addendum: inline functions

Static inline functions are also not part of the introspectable ABI of a library, because they cannot be used with dlsym(); you can provide inlined functions for performance reasons, but remember to always provide their non-inlined equivalent.

Direct C structure access for objects

Again, another fairly uncontroversial rule. You shouldn’t be putting anything into an instance structure, as it makes your API harder to future-proof, and direct access cannot do things like change notification, or memoization.

Always provide accessor functions.

`va_list`

Variadic argument functions are mainly C convenience. Yes, some languages can support them, but it’s a bad idea to have this kind of API exposed as the only way to do things.

Any variadic argument function should have two additional variants:

a vector based version, using C arrays (zero terminated, or with an explicit length)
a va_list version, to be used when creating wrappers with variadic arguments themselves

The va_list variant is kind of optional, since not many people go around writing variadic argument C wrappers, these days, but at the end of the day you might be going to write an internal function that takes a va_list anyway, so it’s not particularly strange to expose it as part of your public API.

The vector-based variant, on the other hand, is fundamental.

Incidentally, if you’re using variadic arguments as a way to collect similarly typed values, e.g.:

// void
// some_object_method (SomeObject *self,
//                     ...) G_GNUC_NULL_TERMINATED

some_object_method (obj, "foo", "bar", "baz", NULL);

there’s very little difference to using a vector and C99’s compound literals:

// void
// some_object_method (SomeObject *self,
//                     const char *args[])

some_object_method (obj, (const char *[]) {
                      "foo",
                      "bar",
                      "baz",
                      NULL,
                    });

Except that now the compiler will be able to do some basic type check and scream at you if you’re doing something egregiously bad.

Compound literals and designated initialisers also help when dealing with key/value pairs:

typedef struct {
  int column;
  union {
    const char *v_str;
    int v_int;
  } value;
} ColumnValue;

enum {
  COLUMN_NAME,
  COLUMN_AGE,
  N_COLUMNS
};

// void
// some_object_method (SomeObject *self,
//                     size_t n_columns,
//                     const ColumnValue values[])

some_object_method (obj, 2,
  (ColumnValue []) {
    { .column = COLUMN_NAME, .data = { .v_str = "Emmanuele" } },
    { .column = COLUMN_AGE, .data = { .v_int = 42 } },
  });

So you should seriously reconsider the amount of variadic arguments convenience functions you expose.

Multiple out parameters

Using a structured type with a out direction is a good recommendation as a way to both limit the amount of out arguments and provide some future-proofing for your API. It’s easy to expand an opaque pointer type with accessors, whereas adding more out arguments requires an ABI break.

Addendum: `inout` arguments

Don’t use in-out arguments. Just don’t.

Pass an in argument to the callable for its input, and take an out argument or a return value for the output.

Memory management and ownership of inout arguments is incredibly hard to capture with static annotations; it mainly works for scalar values, so:

void
some_object_update_matrix (SomeObject *self,
                           double *xx,
                           double *yy,
                           double *xy,
                           double *yx)

can work with xx, yy, xy, yx as inout arguments, because there’s no ownership transfer; but as soon as you start throwing things in like pointers to structures, or vectors of string, you open yourself to questions like:

who allocates the argument when it goes in?
who is responsible for freeing the argument when it comes out?
what happens if the function frees the argument in the in direction and then re-allocates the out?
what happens if the function uses a different allocator than the one used by the caller?
what happens if the function has to allocate more memory?
what happens if the function modifies the argument and frees memory?

Even if gobject-introspection nailed down the rules, they could not be enforced, or validated, and could lead to leaks or, worse, crashes.

So, once again: don’t use inout arguments. If your API already exposes inout arguments, especially for non-scalar types, consider deprecations and adding new entry points.

Addendum: `GValue`

Sadly, GValue is one of the most notable cases of inout abuse. The oldest parts of the GNOME stack use GValue in a way that requires inout annotations because they expect the caller to:

initialise a GValue with the desired type
pass the address of the value
let the function fill the value

The caller is then left with calling g_value_unset() in order to free the resources associated with a GValue. This means that you’re passing an initialised value to a callable, the callable will do something to it (which may or may not even entail re-allocating the value) and then you’re going to get it back at the same address.

It would be a lot easier if the API left the job of initialising the GValue to the callee; then functions could annotate the GValue argument with out and caller-allocates=1. This would leave the ownership to the caller, and remove a whole lot of uncertainty.

Various new (comparatively speaking) API allow the caller to pass an unitialised GValue, and will leave initialisation to the callee, which is how it should be, but this kind of change isn’t always possible in a backward compatible way.

Arrays

You can use three types of C arrays in your API:

zero-terminated arrays, which are the easiest to use, especially for pointers and strings
fixed-size arrays
arrays with length arguments

Addendum: strings and byte arrays

A const char* argument for C strings with a length argument is not an array:

/**
 * some_object_load_data:
 * @self: ...
 * @str: the data to load
 * @len: length of @str in bytes, or -1
 *
 * ...
 */
void
some_object_load_data (SomeObject *self,
                       const char *str,
                       ssize_t len)

Never annotate the str argument with array length=len. Ideally, this kind of function should not exist in the first place. You should always use const char* for NUL-terminated strings, possibly UTF-8 encoded; if you allow embedded NUL characters then use a bytes array:

/**
 * some_object_load_data:
 * @self: ...
 * @data: (array length=len) (element-type uint8): the data to load
 * @len: the length of the data in bytes
 *
 * ...
 */
void
some_object_load_data (SomeObject *self,
                       const unsigned char *data,
                       size_t len)

Instead of unsigned char you can also use uint8_t, just to drive the point home.

Yes, it’s slightly nicer to have a single entry point for strings and byte arrays, but that’s just a C convenience: decent languages will have a proper string type, which always comes with a length; and string types are not binary data.

Addendum: `GArray`, `GPtrArray`, `GByteArray`

Whatever you do, however low you feel on the day, whatever particular tragedy befell your family at some point, please: never use GLib array types in your API. Nothing good will ever come of it, and you’ll just spend your days regretting this choice.

Yes: gobject-introspection transparently converts between GLib array types and C types, to the point of allowing you to annotate the contents of the array. The problem is that that information is static, and only exists at the introspection level. There’s nothing that prevents you from putting other random data into a GPtrArray, as long as it’s pointer-sized. There’s nothing that prevents a version of a library from saying that you own the data inside a GArray, and have the next version assign a clear function to the array to avoid leaking it all over the place on error conditions, or when using g_autoptr.

Adding support for GLib array types in the introspection was a well-intentioned mistake that worked in very specific cases—for instance, in a library that is private to an application. Any well-behaved, well-designed general purpose library should not expose this kind of API to its consumers.

You should use GArray, GPtrArray, and GByteArray internally; they are good types, and remove a lot of the pain of dealing with C arrays. Those types should never be exposed at the API boundary: always convert them to C arrays, or wrap them into your own data types, with proper argument validation and ownership rules.

Addendum: `GHashTable`

What’s worse than a type that contains data with unclear ownership rules decided at run time? A type that contains twice the amount of data with unclear ownership rules decided at run time.

Just like the GLib array types, hash tables should be used but never directly exposed to consumers of an API.

Addendum: `GList`, `GSList`, `GQueue`

See above, re: pain and misery. On top of that, linked lists are a terrible data type that people should rarely, if ever, use in the first place.

Callbacks

Your callbacks should always be in the form of a simple callable with a data argument:

typedef void (* SomeCallback) (SomeObject *obj,
                               gpointer data);

Any function that takes a callback should also take a “user data” argument that will be passed as is to the callback:

// scope: call; the callback data is valid until the
// function returns
void
some_object_do_stuff_immediately (SomeObject *self,
                                  SomeCallback callback,
                                  gpointer data);

// scope: notify; the callback data is valid until the
// notify function gets called
void
some_object_do_stuff_with_a_delay (SomeObject *self,
                                   SomeCallback callback,
                                   gpointer data,
                                   GDestroyNotify notify);

// scope: async; the callback data is valid until the async
// callback is called
void
some_object_do_stuff_but_async (SomeObject *self,
                                GCancellable *cancellable,
                                GAsyncReadyCallback callback,
                                gpointer data);

// not pictured here: scope forever; the data is valid fori
// the entirety of the process lifetime

If your function takes more than one callback argument, you should make sure that it also takes a different user data for each callback, and that the lifetime of the callbacks are well defined. The alternative is to use GClosure instead of a simple C function pointer—but that comes at a cost of GValue marshalling, so the recommendation is to stick with one callback per function.

Addendum: the `closure` annotation

It seems that many people are unclear about the closure annotation.

Whenever you’re describing a function that takes a callback, you should always annotate the callback argument with the argument that contains the user data using the (closure argument) annotation, e.g.

/**
 * some_object_do_stuff_immediately:
 * @self: ...
 * @callback: (scope call) (closure data): the callback
 * @data: the data to be passed to the @callback
 *
 * ...
 */

You should not annotate the data argument with a unary (closure).

The unary (closure) is meant to be used when annotating the callback type:

/**
 * SomeCallback:
 * @self: ...
 * @data: (closure): ...
 *
 * ...
 */
typedef void (* SomeCallback) (SomeObject *self,
                               gpointer data);

Yes, it’s confusing, I know.

Sadly, the introspection parser isn’t very clear about this, but in the future it will emit a warning if it finds a unary closure on anything that isn’t a callback type.

Ideally, you don’t really need to annotate anything when you call your argument user_data, but it does not hurt to be explicit.

A cleaned up version of this blog post will go up on the gobject-introspection website, and we should really have a proper set of best API design practices on the Developer Documentation website by now; nevertheless, I do hope people will actually follow these recommendations at some point, and that they will be prepared for new recommendations in the future. Only dead and unmaintained projects don’t change, after all, and I expect the GNOME stack to last a bit longer than the 25 years it already spans today.

On PyGObject

2022-12-02T18:03:00+00:00

Okay, I can’t believe I have to do this again.

This time, let’s leave the Hellsite out of it, and let’s try to be nuanced from the start. I’d like to avoid getting grief online.

The current state of the Python bindings for GObject-based libraries is making it really hard to recommend using Python as a language for developing GTK and GNOME applications.

PyGObject is currently undermaintained, even after the heroic efforts of Christoph Reiter to keep the fires burning through the long night. The Python community needs more people to work on the bindings, if we want Python to be a first class citizen of the ecosystem.

There’s a lot to do, and not nearly enough people left to do it.

Case study: typed instances

Yes, thou shall use GObject should be the law of the land; but there are legitimate reasons to use typed instances, and GTK 4 has a few of them:

At this very moment, it is impossible to use the types above from Python. PyGObject will literally error out if you try to do so. There are technical reasons why that was a reasonable choice 15+ years ago, but most language bindings written since then can handle typed instances just fine. In fact, PyGObject does handle them, since GParamSpec is a GTypeInstance; of course, that’s because PyGObject has some ad hoc code for them.

Dealing with events and render nodes is not so important in GTK4; but not having access to the expressions API makes writing list widgets incredibly more complicated, requiring to set up everything through UI definition files and never modifying the objects programmatically.

Case study: constructing and disposing

While most of the API in PyGObject is built through introspection, the base wrapper for the GObject class is still very much written in CPython. This requires, among other things, wiring the class’s virtual functions manually, and figuring out the interactions between Python, GObject, and the type system wrappers. This means Python types that inherit from GObject don’t have automatic access to the GObjectClass.constructed and GObjectClass.dispose virtual functions. Normally, this would not be an issue, but modern GObject-based libraries have started to depend on being able to control construction and destruction sequences.

For instance, it is necessary for any type that inherits from GtkWidget to ensure that all its child widgets are disposed manually. While that was possible through the “destroy” signal in GTK3, in GTK4 the signal was removed and everything should go through the GObjectClass.dispose virtual function. Since PyGObject does not allow overriding or implementing that virtual function, your Python class cannot inherit from GtkWidget, but must inherit from ancillary classes like GtkBox or AdwBin. That’s even more relevant for the disposal of resources created through the composite template API. Theoretically, using Gtk.Template would take care of this, but since we cannot plug the Python code into the underlying CPython, we’re stuck.

Case study: documentation, examples, and tutorials

While I have been trying to write Python examples for the GNOME developers documentation, I cannot write everything by myself. Plus, I can’t convince Google to only link to what I write. The result is that searching for “Python” and “GNOME” or “GTK” will inevitably lead people to the GTK3 tutorials and references.

The fragmentation of the documentation is also an issue. The PyGObject website is off to readthedocs.org, which is understandable for a Python projects; then we have the GTK3 tutorial, which hasn’t been updated in a while, and for which there are no plans to have a GTK 4 version.

Additionally, the Python API reference is currently stuck on GTK3, with Python references for GTK4 and friends off on the side.

It would be great to unify the reference sites, and possibly have them under the GNOME infrastructure; but at this point, I’d settle for having a single place to find everything, like GJS did.

What can you do?

Pick up PyGObject. Learn how it works, and how the GObject and CPython API interact.

Write Python overrides to ensure that API like GTK and GIO are nice to use with idiomatic Python.

Write tests for PyGObject.

Write documentation.

Port existing tutorials, examples, and demos to GTK4.

Help Christoph out when it comes to triaging issues, and reviewing merge requests.

Join GNOME Python on Matrix, or the #gnome-python channel on Libera.

Ask questions and provide answers on Discourse.

What happens if nobody does anything?

Right now, we’re in maintenance mode. Things work because of inertia, and because nobody is really pushing the bindings outside of their existing functionality.

Let’s be positive, for a change, and assume that people will show up. They did for Vala when I ranted about it five years ago after a particularly frustrating week dealing with constant build failures in GNOME, so maybe the magic will happen again.

If people do not show up, though, what will likely happen is that Python will just fall on the wayside inside GNOME. Python developers won’t use GTK to write GUI applications; existing Python applications targeting GNOME will either wither and die, or get ported to other languages. The Python bindings themselves may stop working with newer versions of Python, which will inevitably lead downstream distributors to jettison the bindings themselves.

We have been through this dance with the C# bindings and Mono, and the GNOME and Mono communities are all the poorer for it, so I’d like to avoid losing another community of talented developers. History does not need to repeat itself.

Amberol

2022-05-25T01:30:00+01:00

In the beginning…

In 1997, I downloaded my first MP3 file. It was linked on a website, and all I had was a 56k modem, so it took me ages to download the nearly 4 megabytes of 128 kbit/s music goodness. Before that file magically appeared on my hard drive, if we exclude a brief dalliance with MOD files, the only music I had on my computer came either in MIDI or in WAV format.

In the nearly 25 years passed since that seminal moment, my music collection has steadily increased in size — to the point that I cannot comfortably keep it in my laptop’s internal storage without cutting into the available space for other stuff and without taking ages when copying it to new machines; and if I had to upload it to a cloud service, I’d end up paying monthly storage fees that would definitely not make me happy. Plus, I like being able to listen to my music without having a network connection — say, when I’m travelling. For these reasons, I have my music collection on a dedicated USB3 drive and on various 128 GB SD cards that I use when travelling, to avoid bumping around a spinning rust drive.

In order to listen to that first MP3 file, I also had to download a music player, and back in 1997 there was this little software called Winamp, which apparently really whipped the llama’s ass. Around that same time I was also dual-booting between Windows and Linux, and, obviously, Linux had its own Winamp clone called x11amp. This means that, since late 1997, I’ve also tested more or less all mainstream, GTK-based Linux music players—xmms, beep, xmms2, Rhythmbox, Muine, Banshee, Lollypop, GNOME Music—and various less mainstream/non-GTK ones—shout out to ma boi mpg123. I also used iTunes on macOS and Windows, but I don’t speak of that.

Turns out that, with the very special exception of Muine, I can’t stand any of them. They are all fairly inefficient when it comes to managing my music collection; or they are barely maintained; or (but, most often, and) they are just iTunes clones—as if cloning iTunes was a worthy goal for anything remotely connected to music, computing, or even human progress in general.

I did enjoy using Banshee, up to a point; it wasn’t overly offensive to my eyes and pointing devices, and had the advantage of being able to minimise its UI without getting in the way. It just bitrotted with the rest of the GNOME 2 platform even before GNOME bumped major version, and it still wasn’t as good as Muine.

A detour: managing a music collection

I’d like to preface this detour with a disclaimer: I am not talking about specific applications; specific technologies/libraries; or specific platforms. Any resemblance to real projects, existing or abandoned, is purely coincidental. Seriously.

Most music management software is, I feel, predicated on the fallacy that the majority of people don’t bother organising their files, and are thus willing to accept a flat storage with complex views built at run time on top of that; while simultaneously being willing to spend a disproportionate amount of time classifying those files—without, of course, using a hierarchical structure. This is a fundamental misunderstanding of human nature.

By way of an example: if we perceive the Universe in a techno-mazdeist struggle between a πνεῦμα which creates fool-proof tools for users; and a φύσις, which creates more and more adept fools; then we can easily see that, for the entirety of history until now, the pneuma has been kicked squarely in the nuts by the physis. In other words: any design or implementation that does not take into account human nature in that particular problem space is bound to fail.

While documents might benefit from additional relations that are not simply inferred by their type or location on the file system, media files do not really have the same constraints. Especially stuff like music or videos. All the tracks of an album are in the same place not because I decided that, but because the artist or the music producers willed it that way; all the episodes of a series are in the same place because of course they are, and they are divided by season because that’s how TV series work; all the episodes of a podcast are in the same place for the same reason, maybe divided by year, or by season. If that structure already exists, then what’s the point of flattening it and then trying to recreate it every time out of thin air with a database query?

The end result of constructing a UI that is just a view on top of a database is that your UI will be indistiguishable from a database design and management tool; which is why all music management software looks very much like Microsoft Access from circa 1997 onwards. Of course you can dress it up however you like, by adding fancy views of album covers, but at the end of the day it’s just an Excel spread sheet that occasionally plays music.

Another side effect of writing a database that contains the metadata of a bunch of files is that you’ll end up changing the database instead of changing the files; you could write the changes to the files, but reconciling the files with the database is a hard problem, and it also assumes you have read-write access to those files. Now that you have locked your users into your own database, switching to a new application becomes harder, unless your users enjoy figuring out what they changed over time.

A few years ago, before backing up everything in three separate storages, I had a catastrophic failure on my primary music hard drive; after recovering most of my data, I realised that a lot of the changes I made in the early years weren’t written out to music files, but were stored in some random SQLite database somewhere. I am still recovering from that particular disaster.

I want my music player to have read-only access to my music. I don’t want anything that isn’t me writing to it. I also don’t want to re-index my whole music collection just because I fixed the metadata of one album, and I don’t want to lose all my changes when I find a better music player.

Another detour: non-local media

Yes, yes: everyone listens to streamed media these days, because media (and software) companies are speed-running Adam Smith’s The Wealth of Nations and have just arrived at the bit about rentier economy. After all, why should they want to get paid once for something, when media conglomerates can “reap where they never sowed, and demand a rent even for its natural produce”.

You know what streaming services don’t like? Custom, third party clients that they can’t control, can’t use for metrics, and can’t use to serve people ads.

You know what cloud services that offer to host music don’t like? Duplicate storage, and service files that may potentially infringe the IP of a very litigious industry. Plus, of course, third party clients that they can’t use to serve you ads, as that’s how they can operate at all, because this is the Darkest Timeline, and adtech is the modern Moloch to which we must sacrifice as many lives as we can.

You may have a music player that streams somebody’s music collection, or even yours if you can accept the remote service making a mess of it, but you’re always a bad IPO or a bad quarterly revenue report away from losing access to everything.

Writing a music player for fun and no profit

For the past few years I’ve been meaning to put some time into writing a music player, mostly for my own amusement; I also had the idea of using this project to learn the Rust programming language. In 2015 I was looking for a way to read the metadata of music files with Rust, but since I couldn’t find anything decent, I ended up writing the Rust bindings for taglib. I kept noodling at this side project for the following years, but I was mostly hitting the limits of GTK3 when it came to dealing with my music collection; every single iteration of the user interface ended up with a GtkTreeView and a replica of iTunes 1.0.

In the meantime, though, the Rust ecosystem got exponentially better, with lots of crates dedicated to parsing music file metadata; GTK4 got released with new list widgets; libadwaita is available to take care of nice UI layouts; and the Rust bindings for GTK have become one of the most well curated and maintained projects in the language bindings ecosystem.

Another few things that happened in the meantime: a pandemic, a year of unemployment, and zero conferences, all of which pushed me to streaming my free and open source software contributions on Twitch, as a way to break the isolation.

So, after spending the first couple of months of 2022 on writing the beginners tutorial for the GNOME developer documentation website, in March I began writing Amberol, a local-only music player that has no plans of becoming more than that.

Desktop mode

Amberol’s scope sits in the same grand tradition of Winamp, and while its UI started off as a Muine rip off—down to the same key shortcuts—it has evolved into something that more closely resembles the music player I have on my phone.

Mobile mode

Amberol’s explicit goal is to let me play music on my desktop the same way I typically do when I am using my phone, which is: shuffling all the songs in my music collection; or, alternatively, listening to all the songs in an album or from an artist from start to finish.

Amberol’s explicit non goals are:

managing your music collection
figuring out your music metadata
building playlists
accessing external services for stuff like cover art, song lyrics, or the artist’s Wikipedia page

The actual main feature of this application is that it has forced me to figure out how to deal with GStreamer after 15 years.

I did try to write this application in a way that reflects the latest best practices of GTK4:

model objects
custom view widgets
composite widgets using templates
property bindings/expressions to couple model/state to its view/representation
actions and actionable widgets

The ability to rely on libadwaita has allowed me to implement the recoloring of the main window without having to deal with breakage coming from rando style sheets:

The main thing I did not expect was how much of a good fit was Rust in all of this. The GTK bindings are top notch, and constantly improving; the type system has helped me much more than hindering me, a poor programmer whose mind has been twisted by nearly two decades of C. Good idiomatic practices for GTK are entirely within the same ballpark of idiomatic practices for Rust, especially for application development.

On the tooling side, Builder has been incredibly helpful in letting me concentrate on the project—starting from the basic template for a GNOME application in Rust, to dealing with the build system; from the Flatpak manifest, to running the application under a debugger. My work was basically ready to be submitted to Flathub from day one. I did have some challenge with the AppData validation, mostly caused by appstream-utils undocumented validation rules, but luckily it’s entirely possible to remove the validation after you deal with the basic manifest.

All in all, I am definitely happy with the results of basically two months of hacking and refactoring, mostly off an on (and with two weeks of COVID in the middle).

Fair Weather Friends

2021-11-19T20:48:00+01:00

Today I released libgweather-3.90.0, the first developers snapshot of GWeather 4:

Behold! A project logo

This release is mostly meant to be used as a target for porting existing code to the new API, and verifying that everything works as it should.

The major changes from GWeather-3.0 are:

the GTK3 widgets have gone to a farm up state, so you’ll have to write your own UI for searching locations
GWeatherLocation is a GObject type, so you can use it with GListModel and friends, which should help with the point above
the deprecated API has been removed
the API that will be part of GWeather 4.0 will be stable, and regular API/ABI stability guarantees will apply

If you are using libgweather in your application, you should head over to the migration guide and check out what changed.

Ideally, there are still things that need to be cleaned up in the GWeather API, for instance:

GWeatherTimezone parses the tzdata file directly, which comes with its own set of issues, like having to track whether the time zone database has changed or not; we should use GTimeZone instead, but the API provided by the two types do not match entirely. I need to check the current users of that API, and if possible, just drop the whole type.
GWeatherInfo has a bunch of getter functions that return the bare values and that can fail, and additional getter functions that always return a formatted string, and cannot fail; it’s not a great API.
GWeatherLocation returns the localised names by default, and has additional getters for the “English” (really: POSIX C locale) names.

If you encounter issues when porting, please: file an issue on GitLab.

GWeather next

2021-10-15T10:52:00+01:00

tl;dr Libgweather, the small GNOME library that queries weather services, is getting a major version bump to allow applications using it to be ported to GTK4.

In the beginning, there was a weather applet in the GNOME panel. It had a bunch of code that poked at a couple of websites to get the weather information for a given airport or weather observation stations, and shipped with a list of locations and their nearest METAR code.

In 2007, the relevant code was moved to its own separate repository, so that other applications and system settings could reuse the same code as the panel applet: the libgweather library was born. Aside from the basic weather information and location objects, libgweather also had a couple of widgets: one for selecting a location (with autocompletion), and one for selecting a timezone using a location.

Since libgweather was still very much an ad hoc library for a handful of applications, there was no explicit API and ABI stability guarantee made by its maintainers; in fact, in order to use it, you had to “opt in” with a specific C pre-processor symbol.

Time passed, and a few more applications appeared during the initial GNOME 3 cycles—like Weather, followed by Clocks a month later. Most of the consumers of libgweather were actually going through a language binding, which meant they were not really “opting into” the API through the explicit pre-processor symbol; it also meant that changes in the API and ABI could end up being found only after a libgweather release, instead of during a development cycle. Of course, back then, we only had a single CI/CD pipeline for the whole project, with far too little granularity and far too wide scope. Still, the GWeather consumers were few and far between, and the API was not stabilised.

Fast forward to now.

The core GNOME applications using GWeather are in the process of being ported to GTK4, but GWeather still ships with two GTK3 widgets. Since you cannot have GTK3 and GTK4 types in the same process, this requires either porting GWeather to GTK4 or dropping the widgets. As it turns out, the widgets are not really shared across applications using libgweather, and all of them have also been redesigned or are using the libadwaita/GTK4 port as a chance to refresh their overall appearences. This makes our life a little bit easier, as we can drop the widgets without really losing any actual functionality that people do care about.

For GNOME 42, the plan for libgweather is:

bump up the API version to 4.0, and ensure parallel installability with the older libgweather-3; this requires renaming things like the pkg-config file and the settings schema, alongside the shared library
drop the GTK widgets, and some old API that hasn’t been working in years, like getting the radar image animation
stabilise the API, and turn libgweather into a proper library, with the usual API and ABI stability guarantees (deprecations and new symbols added only during development cycles, no changes/removals until the following major API bump)
make it easier to use libgweather objects with GListModel-based API
document the API properly
clean up the internals from various years of inconsistent coding style and practices

I’m also going through the issues imported from Bugzilla and closing the ones that have long since been fixed.

In the meantime, the old libgweather-3 API is going to be frozen, for the tools that still use it and won’t be ported to GTK4 any time soon.

For more information, you can read:

If you’re using libgweather, I strongly recommend you to use the 40.0 release or build from the libgweather-3 branch until you are planning to port to GTK4.

If you’re distributing libgweather, I recommend you package the new libgweather under a new name, given that it’s parallel installable with the old one; my recommendation is to use libgweather4 or libgweather-4 as the name of the package.

Properties, introspection, and you

2021-09-21T21:22:00+01:00

It is a truth universally acknowledged, that a GObject class in possession of a property, must be in want of an accessor function.

The main issue with that statement is that it’s really hard to pair the GObject property with the accessor functions that set the property’s value, and retrieve it.

From a documentation perspective, tools might not establish any relation (gtk-doc), or they might require some additional annotation to do so (gi-docgen); but at the introspection level there’s nothing in the XML or the binary data that lets you go from a property name to a setter, or a getter, function. At least, until now.

GObject-introspection 1.70, released alongside GLib 2.70 and GNOME 41, introduced various annotations for both properties and methods that let you go from one to the other; additionally, new API was added to libgirepository to allow bindings to dynamic languages to establish that relation at run time.

Annotations

If you have a property, and you document it as you should, you’ll have something like this:

/**
 * YourWidget:your-property
 *
 * A property that does something amazing.
 */

If you want to associate the setter and getter functions to this property, all you need to do is add the following identifier annotations to it:

/**
 * YourWidget:your-property: (setter set_your_property) (getter get_your_property)
 *
 * A property that does something amazing.
 */

The (setter) and (getter) annotations take the name of the method that is used to set, and get, the property, respectively. The method name is relative to the type, so you should not pass the C symbol.

On the accessor methods side, you have two additional annotations:

/**
 * your_widget_set_your_property: (set-property your-property)
 * @self: your widget
 * @value: the value to set
 *
 * Sets the given value for your property.
 */

and:

/**
 * your_widget_get_your_property: (get-property your-property)
 * @self: your widget
 *
 * Retrieves the value of your property.
 *
 * Returns: the value of the property
 */

Heuristics

Of course, you’re now tempted to go and add those annotations to all your properties and related accessors. Before you do that, though, you should know that the introspection scanner will try and match properties and accessors by itself, using appropriate heuristics:

if your object type has a writable, non-construct-only property, and a method that is called set_<property>, then the property will have a setter and the method will be matched to the property
if your object type has a readable property, and a method that is called get_<property>, then the property will have a getter and the method will be matched to the property
additionally, if the property is read-only and the property type is boolean, the scanner will look at a method that has the same name as the property as well; this is meant to catch getters like gtk_widget_has_focus(), which accesses the read-only property has-focus

API

All of the above ends up in the introspection XML, which is used by documentation tools and code generators. Bindings for dynamic languages using libgirepository can also access this information at run time, by using the API in GIPropertyInfo to retrieve the setter and getter function information for a property; and the API in GIFunctionInfo to retrieve the property being set.

Future

Ideally, with this information, language bindings should be able to call the accessor functions instead of going through the generic g_object_set_property() and g_object_get_property() API, except as a fallback. This should speed up the property access in various cases. Additionally, bindings could decide to stop exposing C accessors, and only expose the property, in order to make the API more idiomatic.

On the documentation side, this will ensure that tools like gi-docgen will be able to bind the properties and their accessors more reliably, without requiring extra attributes.

And one more thing

One thing that did not make it in time for the 1.70 release, but will land early in the next development cycle for gobject-introspection, is the validation for properties. Language bindings don’t really like it when the C API exposes properties that have the same name of methods and virtual functions; we already have a validation pass ready to land, so expect warnings in the near future.

Another feature that will land early in the cycle is the (emitter) annotation, which will bind a method emitting a signal with the signal name. This is a feature taken from Vala’s metadata, and should improve the quality of life of people using introspection data with Vala, as well as removing the need for another attribute in gi-docgen.

Finally, if you maintain a language binding: please look at !204, and make sure you’re not calling g_assert_not_reached() or g_error() when encountering a new scope type. The forever scope cannot land if it breaks every single binding in existence.

Publishing your documentation

2021-08-26T00:51:00+01:00

The main function of library-web, the tool that published the API reference of the various GNOME libraries, was to take release archives and put their contents in a location that would be visible to a web server. In 2006, this was the apex of automation, of course. These days? Not so much.

Since library-web is going the way of the Dodo, and we do have better ways to automate the build and publishing of files with GitLab, how do we replace library-web in 2021? The answer is, unsurprisingly: continuous integration pipelines.

I will assume that you’re already building—and testing—your library using GitLab’s CI; if you aren’t, then you have bigger problems than just publishing your API.

So, let’s start with these preconditions:

your project is hosted on GNOME’s GitLab instance
your project is using Meson
your project is using gtk-doc or gi-docgen

If your project doesn’t satisfy these preconditions you might want to work on doing so; alternatively, you can implement your own CI pipeline.

Let’s start with a simple job template:

# Expected variables:
# PROJECT_DEPS: the dependencies for your own project
# MESON_VERSION: the version of Meson you depend on
# MESON_EXTRA_FLAGS: additional Meson setup options
#   you wish to pass to the configuration phase
# DOCS_FLAGS: the Meson setup option for enabling the
#   documentation, if any
# DOCS_PATH: the path of the generated reference,
#   relative to the build root
.gidocgen-build:
  image: fedora:latest
  before_script:
    - export PATH="$HOME/.local/bin:$PATH"
    - >
      dnf install -y
      python3
      python3-pip
      python3-wheel
      gobject-introspection-devel
      graphviz
      ninja-build
      redhat-rpm-config
    - dnf install -y ${PROJECT_DEPS}
    - >
      pip3 install
      meson==${MESON_VERSION}
      gi-docgen
      jinja2
      Markdown
      markupsafe
      pygments
      toml
      typogrify
  script:
    - meson setup ${MESON_EXTRA_FLAGS} ${DOCS_FLAGS} _docs .
    - meson compile -C _docs
    - |
      pushd "_docs/${DOCS_PATH}" > /dev/null 
      tar cfJ ${CI_PROJECT_NAME}-docs.tar.xz .
      popd > /dev/null
    - mv _docs/${DOCS_PATH}/${CI_PROJECT_NAME}-docs.tar .
  artifacts:
    when: always
    name: 'Documentation'
    expose_as: 'Download the API reference'
    paths:
      - ${CI_PROJECT_NAME}-docs.tar.xz

This CI template will:

download all the required dependencies for building the API reference using gi-docgen
build your project, including the API reference
create an archive with the API reference
store the archive as a CI artefact that you can easily download

Incidentally, by adding a meson test -C _build to the script section, you can easily test your build as well; and if you have a test() target in your build that runs gi-docgen check, then you can verify that your documentation is always complete.

Now, all you have to do is create your own CI job that inherits from the template inside its own stage. I will use JSON-GLib as a reference:

stages:
  - docs

api-reference:
  stage: docs
  extends: .gidocgen-build
  needs: []
  variables:
    MESON_VERSION: 0.55.3
    DOCS_FLAGS: -Dgtk_doc=true
    PROJECT_DEPS:
      gcc
      git
      gettext
      glib2-devel
    DOCS_PATH: docs/json-glib-1.0

“What about gtk-doc!”, I hear from the back of the room. Well, fear not, because there’s a similar template you can use if you’re still using gtk-doc in your project:

# Expected variables:
# PROJECT_DEPS: the dependencies for your own project
# MESON_VERSION: the version of Meson you depend on
# MESON_EXTRA_FLAGS: additional Meson setup options you
#   wish to pass to the configuration phase
# DOCS_FLAGS: the Meson setup option for enabling the
#   documentation, if any
# DOCS_TARGET: the Meson target for building the
#   documentation, if any
# DOCS_PATH: the path of the generated reference,
#   relative to the build root
.gtkdoc-build:
  image: fedora:latest
  before_script:
    - export PATH="$HOME/.local/bin:$PATH"
    - >
      dnf install -y
      python3
      python3-pip
      python3-wheel
      gtk-doc
      ninja-build
      redhat-rpm-config
    - dnf install -y ${PROJECT_DEPS}
    - pip3 install  meson==${MESON_VERSION}
  script:
    - meson setup ${MESON_EXTRA_FLAGS} ${DOCS_FLAGS} _docs .
    # This is exceedingly annoying, but sadly its how
    # gtk-doc works in Meson
    - ninja -C _docs ${DOCS_TARGET}
    - |
      pushd "_docs/${DOCS_PATH}" > /dev/null 
      tar cfJ ${CI_PROJECT_NAME}-docs.tar.xz .
      popd > /dev/null
    - mv _docs/${DOCS_PATH}/${CI_PROJECT_NAME}-docs.tar .
  artifacts:
    when: always
    name: 'Documentation'
    expose_as: 'Download the API reference'
    paths:
      - ${CI_PROJECT_NAME}-docs.tar.xz

And now you can use extends: .gtkdoc-build in your api-reference job.

Of course, this is just half of the job: the actual goal is to publish the documentation using GitLab’s Pages. For that, you will need another CI job in your pipeline, this time using the deploy stage:

stages:
  - docs
  - deploy

# ... the api-reference job goes here...

pages:
  stage: deploy
  needs: ['api-reference']
  script:
    - mkdir public && cd public
    - tar xfJ ../${CI_PROJECT_NAME}-docs.tar.xz
  artifacts:
    paths:
      - public
  only:
    - master
    - main

Now, once you push to your main development branch, your API reference will be built by your CI pipeline, and the results published in your project’s Pages space—like JSON-GLib.

The CI pipeline and GitLab Pages are also useful for building complex, static websites presenting multiple versions of the documentation; or presenting multiple libraries. An example of the former is libadwaita’s website, while an example of the latter is the GTK documentation website. I’ll write a blog post about them another time.

Given that the CI templates are pretty generic, I’m working on adding them into the GNOME ci-templates repository, so you will be able to use something like:

include: 'https://gitlab.gnome.org/GNOME/citemplates/raw/HEAD/docs/gidocgen.yml'

or:

include: 'https://gitlab.gnome.org/GNOME/citemplates/raw/HEAD/docs/gtkdoc.yml'

without having to copy-paste the template in your own .gitlab-ci.yml file.

The obvious limitation of this approach is that you will need to depend on the latest version of Fedora to build your project. Sadly, we cannot use Flatpak and the GNOME run time images for this, mainly because we are building libraries, not applications; and because extracting files out of a Flatpak build after it has completed isn’t entirely trivial. Another side effect is that if you bump up the dependencies of your project to something on the bleeding edge and currently not packaged on the latest stable Fedora, you will need to have it included as a Meson sub-project. Of course, you should already be doing that, so it’s a minor downside.

Ideally, if GNOME built actual run time images for the SDK, we could install gtk-doc, gi-docgen, and all their dependencies into the SDK itself, and avoid depending on a real Linux distribution for the libraries in our platform.

Documenting GNOME for developers

2021-08-01T19:31:00+01:00

You may have just now noticed that the GNOME developers documentation website has changed after 15 years. You may also have noticed that it contains drastically less content than it used to. Before you pick up torches and pitchforks, let me give you a short tl;dr of the changes:

Yes, this is entirely intentional
Yes, I know that stuff has been moved
Yes, I know that old URLs don’t work
Yes, some redirections will be put in place
No, we can’t go back

So let’s recap a bit the state of the developers documentation website in 2021, for those who weren’t in attendance at my GUADEC 2021 presentation:

library-web is a Python application, which started as a Summer of Code project in 2006, whose job was to take Autotools release tarballs, explode them, fiddle with their contents, and then publish files on the gnome.org infrastructure.
library-web relies heavily on Autotools and gtk-doc.
library-web does a lot of pre-processing of the documentation to rewrite links and CSS from the HTML files it receives.
library-web is very much a locally sourced, organic, artisanal pile of hacks that revolve very much around the GNOME infrastructure from around 2007-2009.
library-web is incredibly hard to test locally, even when running inside a container, and the logging is virtually non-existent.
library-web is still running on Python 2.
library-web is entirely unmaintained.

That should cover the infrastructure side of things. Now let’s look at the content.

The developers documentation is divided in four sections:

a platform overview
the Human Interface guidelines
guides and tutorials
API references

The platform overview is slightly out of date; the design team has been reviewing the HIG and using a new documentation format; the guides and tutorials still like GTK1 and GTK2 content; or how to port GNOME 2 applications to GNOME 3; or how to write a Metacity theme.

This leaves us with the API references, which are a grab bag of miscellaneous things, listed by version numbers. Outside of the C API documentation, the only other references hosted on developer.gnome.org are the C++ bindings—which, incidentally, use Doxygen and when they aren’t broken by library-web messing about with the HTML, they have their own franken-style mash up of gtkmm.org and developer.gnome.org.

Why didn’t I know about this?

If you’re asking this question, allow me to be blunt for a second: the reason you never noticed that the developers documentation website was broken is that you never actually experienced it for its intended use case. Most likely, you either just looked in a couple of well known places and never ventured outside of those; and/or you are a maintainer, and you never literally cared how things worked (or didn’t work) after you uploaded a release tarball somewhere. Like all infrastructure, it was somebody else’s problem.

I completely understand that we’re all volunteers, and that things that work can be ignored because everyone has more important things to think about.

Sadly, things change: we don’t use Autotools (that much), which means release archives do not contain the generated documentation any more; this means library-web cannot be updated, unless somebody modifies the configuration to look for a separate documentation tarball that the maintainer has to generate manually and upload in a magic location on the gnome.org file server—this has happened for GTK4 and GLib for the past two years.

Projects change the way they lay out the documentation, or gtk-doc changes something, and that causes library-web to stop extracting the right files; you can look at the ATK reference for the past year and a half for an example.

Projects bump up their API, and now the cross-referencing gets broken, like the GTK3 pages linking GDK2 types.

Finally, projects decide to change how their documentation is generated, which means that library-web has no idea how to extract the HTML files, or how to fiddle with them.

If you’re still using Autotools and gtk-doc, and haven’t done an API bump in 15 years, and all you care about is copying a release archive to the gnome.org infrastructure I’m sure all of this will come as a surprise, and I’m sorry you’re just now being confronted with a completely broken infrastructure. Sadly, the infrastructure was broken for everybody else long before this point.

What did you do?

I tried to make library-web deal with the changes in our infrastructure. I personally built and uploaded multiple versions of the documentation for GLib (three different archives for each release) for a year and a half; I configured library-web to add more “extra tarball” locations for various projects; I tried making library-web understand the new layout of various projects; I even tried making library-web publish the gi-docgen references used by GTK, Pango, and other projects.

Sadly, every change broke something else—and I’m not just talking about the horrors of the code base. As library-web is responsible for determining the structure of the documentation, any change to how the documentation is handled leads to broken URLs, broken links, or broken redirections.

The entire castle of cards needed to go.

Which brings us to the plan.

What are you going to do?

Well, the first step has been made: the new developer.gnome.org website does not use library-web. The content has been refreshed, and more content is on the way.

Again, this leaves the API references. For those, there are two things that need to happen—and are planned for GNOME 41:

all the libraries that are part of the GNOME SDK run time, built by gnome-build-meta must also build their documentation, which will be published as part of the org.gnome.Sdk.Docs extension; the contents of the extension will also be published online.
every library that is hosted on gnome.org infrastructure should publish their documentation through their CI pipeline; for that, I’m working on a CI template file and image that should take care of the easy projects, and will act as model for projects that are more complicated.

I’m happy to guide maintainers to deal with that, and I’m also happy to open merge requests on various projects.

In the meantime, the old documentation is still available as a static snapshot, and the sysadmins are going to set up some redirections to bridge us from the old platform to the new—and hopefully we’ll soon be able to redirect to each project’s GitLab pages.

Can we go back, please?

Sadly, since nobody has ever bothered picking up the developers documentation when it was still possible to incrementally fix it, going back to a broken infrastructure isn’t going to help anybody.

We also cannot keep the old developer.gnome.org and add a new one, of course; now we’d have two websites, one of which broken and unmaintained and linked all over the place, and a new one that nobody knows exists.

The only way is forward, for better or worse.

What about Devhelp

Some of you may have noticed that I picked up the maintenance of Devhelp, and landed a few fixes to ensure that it can read the GTK4 documentation. Outside of some visual refresh for the UI, I also am working on making it load the contents of the org.gnome.Sdk.Docs run time extension, which means it’ll be able to load all the core API references. Ideally, we’re also going to see a port to GTK4 and libadwaita, as soon as WebKitGTK for GTK4 is more wideley available.

Final Types

2021-07-27T17:30:00+01:00

The type system at the base of our platform, GType, has various kinds of derivability:

simple derivability, where you’re allowed to create your derived version of an existing type, but you cannot derive your type any further;
deep derivability, where you’re allowed to derive types from other types;

An example of the first kind is any type inheriting from GBoxed, whereas an example of the second kind is anything that inherits from GTypeInstance, like GObject.

Additionally, any derivable type can be marked as abstract; an abstract type cannot be instantiated, but you can create your own derived type which may or may not be “concrete”. Looking at the GType reference documentation, you’ll notice various macros and flags that exist to implement this functionality—including macros that were introduced to cut down the boilerplate necessary to declare and define new types.

The G_DECLARE_* family of macros, though, introduced a new concept in the type system: a “final” type. Final types are leaf nodes in the type hierarchy: they can be instantiated, but they cannot be derived any further. GTK 4 makes use of this kind of types to nudge developers towards composition, instead of inheritance. The main problem is that the concept of a “final” type is entirely orthogonal to the type system; there’s no way to programmatically know that a type is “final”—unless you have access to the introspection data and start playing with heuristics about symbol visibility. This means that language bindings are unable to know without human intervention if a type can actually be inherited from or not.

In GLib 2.70 we finally plugged the hole in the type system, and we introduced the G_TYPE_FLAG_FINAL flag. Types defined as “final” cannot be derived any further: as soon as you attempt to register your new type that inherits from a “final” type, you’ll get a warning at run time. There are macros available that will let you define final types, as well.

Thanks to the “final” flag, we can also include this information into the introspection data; this will allow language bindings to warn you if you attempt at inheriting from a “final” type, likely using language-native tools, instead of getting a run time warning.

If you are using G_DECLARE_FINAL_TYPE in your code you should bump up your GObject dependency to 2.70, and switch your implementation from G_DEFINE_TYPE and friends to G_DEFINE_FINAL_TYPE.

Documentation changes

2021-02-19T21:16:00+00:00

Back in the late ‘90s, people working on GTK had the exact same problem we have today: how do we document the collection of functions, types, macros, and assorted symbols that we call “an API”. It’s all well and good to strive for an API that can be immediately grasped by adhering to a set of well defined conventions and naming; but nothing is, or really can be, “self documenting”.

When GTK 1.0 was released, the documentation was literally stored in handwritten Texinfo files; the API footprint of GTK was small enough to still make it possible, but not really maintainable in the longer term. In 1998, a new system was devised for documenting GTK 1.2:

a script, to parse the source files for the various declarations and dump them into machine parseable “templates”, that would then be modified to include the actual documentation, and committed to the source repository
a small tool that would generate, compile, and run a small tool to introspect the type system for things like hierarchy and signals
a script to take the templates, the list of symbols divided into logical “sections”, an index file, and generate a bunch of DocBook XML files
finally, a script to convert DocBook to HTML or man pages, via xsltproc and an XML stylesheet

Whenever somebody added a new symbol to GTK, they would need to run the script, find the symbol in the template files, write the documentation using DocBook tags if necessary, and then commit the changes alongside the rest of the code.

Since this was 1998, and the scripts had to parse a bunch of text files using regular expressions, they were written in Perl and strung together with a bunch of Makefile rules.

Thus, gtk-doc was born.

Of course, since other libraries needed to provide an API reference to those poor souls using them, gtk-doc ended up being shared across the GNOME platform. We even built part of our website and release infrastructure around it.

At some point between 1998 and 2009, gtk-doc gained the ability to generate those template files incrementally, straight from the C sources; this allowed moving the preambles of each section into the corresponding source file, thus removing the templates from the repository, and keeping the documentation close to the code it references, in the hope it would lead to fewer instances of docs drift.

Between 2009 and 2021, a few things happened:

gobject-introspection has become “a thing”; g-i also parses the C code, and does it slightly more thoroughly than gtk-doc, to gather the same information: declarations, hierarchy, interfaces, properties, and signals, and even documentation, which is all shoved into a well-defined XML file; on top of that, g-i needs annotations in the source to produce a machine-readable description of the C ABI of a library, which can then be used to generate language bindings
turns out that DocBook is pretty terrible, and running xsltproc on large, complex DocBook files is really slow
Perl isn’t really a Hot Language™ like it was in the late ‘90s; many Linux distributions dropped it from the core installation, and not many people speak it that fluently, which means not many people will want to help with a large Perl application

To cope with issue (1), gtk-doc had to learn to parse introspection annotations.

Issue (2) led to replacing DocBook tags inside the inline documentation with subset of Markdown, augmented with custom code blocks and intra-document anchors for specific sections.

Issue (3) led to a wholesale rewrite in Python, in the hope that more people would contribute to the maintenance of gtk-doc.

Sadly, all three solutions ended up breaking things in different ways:

gtk-doc never really managed to express the introspection information in the generated documentation, outside of references to an ancillary appendix. If an annotation says “this argument can be NULL”, for instance, there’s no need to write “or NULL” in the documentation itself: the documentation tool can write it out for you.
the move to Markdown means that existing DocBook tags in the documentation are now ignored or, worse, misinterpreted for HTML and not rendered; this requires porting all the documentation in every library, in a giant flag day, to avoid broken docs; on top of that, DocBook’s style sheet to generate HTML started exhibiting regressions after a build system change, which led, among other things, to the disappearance of per-version symbols indices
the port to Python probably came too late, and ended up having many, many regressions; gtk-doc is still a pretty complex tool, and it still caters to many different use cases, spanning two decades; as much as its use is documented and tested, its internals are really not, meaning that it’s not an easy project to pick up

Over the past 10 years various projects started migrating away from gtk-doc; gobject-introspection itself shipped a documentation tool capable of generating API references, though it mostly is a demonstrator of potential capabilities more than an actual tool. Language bindings, on the other hand, adopted the introspection data as the source for their documentation, and you can see it in Python, JavaScript, and Rust.

As much as I’d like to contribute to gtk-doc, I’m afraid we reached the point where we might want to experiment with something more radical, instead of patching something up, and end up breaking what’s left.

So, since we’re starting from the bottom up, let’s figure out what are the requirements for a tool to generate the documentation for GTK:

be fast. Building GTK’s API reference takes a long time. The API footprint of GDK, GSK, and GTK is not small, but there’s no reason why building the documentation should take a comparable amount of time as building the library. We moved to Meson because it has improved the build times of GTK, we don’t want to get into a bottleneck now.
no additional source parsing. We already parse the C sources in order to generate the introspection data, we don’t need another pass at that.
tailored for GTK. Whenever GTK changes, the tool must change with GTK; the output must adapt to the style of documentation GTK uses.
integrated with GTK. We don’t want an external dependency that makes it harder to deploy the GTK documentation on non-Linux platforms. Using it as a sub-project would be the best option, followed by being able to install it everywhere without additional, Linux-only dependencies.

The explicit non-goal is to create a general purpose documentation tool. We don’t need that; in fact: we’re actively avoiding it. Regardless of what you’ve been taught at university, or your geeky instincts tell you, not every problem requires a generic solution. The whole reason why we are in this mess is that we took a tool for generating the GTK documentation and then generalised the approach until it fell apart under its own weight.

If you want a general purpose documentation tool for C and C++ libraries, there are many to choose from:

There’s also gtk-doc: if you’re using it already, I strongly recommend helping out with its maintenance.

Back in November 2020, as a side project while we were closing in to the GTK 4.0 release date, I started exploring the idea of parsing the introspection data to generate the C API reference for GTK. I wanted to start from scratch, and see how far I could go, so I deliberately avoided taking the GIR parser from gobject-introspection; armed only with the GIR schema and a bunch of Python, I ended up writing a decent parser that would be able to load the GTK introspection XML data, including its dependencies, and dump the whole tree of C identifiers and symbols. After a break, at the end of January 2021, I decided to take a page out the static website generator rule book, and plugged the Jinja templates into the introspection data. The whole thing took about a couple of weeks to go from this:

Everybody loves a tree-like output on the command line

to this:

Behold! My stuff!

My evil plan of generating something decent enough to be usable and then showing it to people with actual taste and web development skills paid off, because I got a whole merge request from Martin Zilz to create a beautiful theme, with support for responsive layout and even for a dark variant:

Amazing what actual taste and skill can accomplish

Like night and day

Turns out that when you stop parsing C files and building small binaries to introspect the type system, and remove DocBook and xsltproc from the pipeline, things get fast. Who knew…

Additionally, once you move the template and the styling outside of the generator, and you can create more complex documentation hierarchies, while retaining the ability for people that are not programmers to change the resulting HTML.

The interesting side effect of using introspection data is that our API reference is now matching what language bindings are able to see and consume—and oh boy, do we suck at that. Part of the fault lies in the introspection parser not being able to cover some of the nastiest parts of C, like macros—though, hopefully, that will improve in the near future; but a lot of issues come from our own API design. Even after 10 years since the introduction of introspection, we’re still doing some very dumb things when it comes to C API—ad let’s ignore stuff that happened 20 years ago and that we haven’t been able to fix yet. Hopefully, the documentation slapping us in the face is going to help us figuring things out before they hit stable releases.

What’s missing from gi-docgen? Well, you can look at the 2021.1 milestone on GitLab:

more documentation on the ancillary files used for project and template configuration; stabilising the key/value pairs would also be part of the documentation effort
client-side search, with symbols exposed to tools like GNOME Builder through something that isn’t quite as tragic as DevHelp files
automatic cross-linking with dependencies documented by gi-docgen, especially for libraries with multiple namespaces, like GTK and Pango
generating proper dependency files, for the consumption of build tools like Meson

In the meantime, what’s missing for GTK to use this? Mainly, porting the documentation away from the various gtk-doc-isms, like marking symbols with sigils, or using |[ ... ]| to define code blocks. Additionally, since the introspection scanner only attaches SECTION blocks to the documentation element of a class, all the sections that operate as “grab bag of related symbols” need to be moved to a separate Markdown file.

It must needs be remarked: gi-docgen is not a generic solution for documenting C libraries. If your library does not have introspection, doesn’t use type classes, or has a very different ABI exposed through introspection than the actual C API, then you’re not going to find it useful—and I don’t have any plans to cater to your use cases either. You should keep using gtk-doc if it still works for you; or you may want to consider other documentation tools.

Anything that complicates the goal of this tool—generating the API reference for GTK and ancillary libraries—is completely out of scope.

Type instances

2020-06-02T16:06:00+01:00

Let us assume we are writing a library.

The particular nature of our work is up for any amount of debate, but the basic fact of it comes with a few requirements, and they are by and large inevitable if you wish to be a well-behaved, well-integrated member of the GNOME community. One of which is: “please, think of the language bindings”. These days, luckily for all of us, this means writing introspectable interfaces that adhere to fairly sensible best practices and conventions.

One of the basic conventions has to do with types. By and large, types exposed by libraries fall into these two categories:

plain old data structures, which are represented by what’s called a “boxed” type; these are simple types with a copy and a free function, mostly meant for marshalling things around so that language bindings can implement properties, signal handlers, and abide to ownership transfer rules. Boxed types cannot have sub-types.
object types, used for everything else: properties, emitting signals, inheritance, interface implementation, the whole shebang.

Boxed and object types cover most of the functionality in a modern, GObject-based API, and people can consume the very same API from languages that are not C.

Except that there’s a third, kind of niche data type:

fully opaque, with instance fields only known within the scope of the project itself
immutable, or at least with low-mutability, after construction
reference counted, with optional cloning and serialization
derivable within the scope of the project, typically with a base abstract class
without signals or properties

Boxing

One strategy used to implement this niche type has been to use a boxed type, and then invent some private, ad hoc derivation technique, with some structure containing function pointers used as a vtable, for instance:

boxed-type.c[Lines 3-79]download

/* {{{ Base */
typedef struct _Base            Base;
typedef struct _BaseClass       BaseClass;

struct _BaseClass
{
  const char *type_name;
  gsize instance_size;

  void  (* finalize)    (Base *self);
  void  (* foo)         (Base *self);
  void  (* bar)         (Base *self);
}

struct _Base
{
  const BaseClass *base_class;

  // Shared field
  int some_field;
};

// Allocate the instance described in the vtable
static Base *
base_alloc (const BaseClass *vtable)
{
  // Simple check to ensure that the derived instance includes the
  // parent base type
  g_assert (vtable->instance_size >= sizeof (Base));

  // Use the atomic refcounted boxing to allocated the requested
  // instance size
  Base *res = g_atomic_rc_box_new (vtable->instance_size);

  // Store the vtable
  res->base_class = vtable;

  // Initialize the base instance fields
  res->some_field = 42;

  return res;
}

static void
base_finalize (Base *self)
{
  // Allow derived types to clear up their own instance data
  self->base_class->finalize (self);
}

Base *
base_ref (Base *self)
{
  return g_atomic_rc_box_acquire (self);
}

void
base_unref (Base *self)
{
  g_atomic_rc_box_release (self, base_finalize);
}

void
base_foo (Base *self)
{
  self->base_class->foo (self);
}

void
base_bar (Base *self)
{
  self->base_class->bar (self);
}

// Add a GType for the base type
G_DEFINE_BOXED_TYPE (Base, base, base_ref, base_unref)
/* }}} */

The code above lets us create derived types that conform to the base type API contract, while providing additional functionality; for instance:

boxed-type.c[Lines 81-123]download

/* {{{ DerivedA */
typedef struct {
  Base parent;

  char *some_other_field;
} DerivedA;

static void
derived_a_finalize (Base *base)
{
  DerivedA *self = (DerivedA *) base;

  g_free (self->some_other_field);
}

static const BaseClass derived_a_class = {
  .type_name = "DerivedA",
  .instance_size = sizeof (DerivedA),
  .finalize = derived_a_finalize,
  .foo = derived_a_foo, // defined elsewhere
  .bar = derived_a_bar, // defined elsewhere
};

Base *
derived_a_new (const char *some_other_field)
{
  Base *res = base_alloc (&derived_a_class);

  DerivedA *self = (DerivedA *) res;

  self->some_other_field = g_strdup (some_other_field);

  return res;
}

const char *
derived_a_get_some_other_field (Base *base)
{
  DerivedA *self = (DerivedA *) base;

  return self->some_other_field;
}
/* }}} */

Since the Base type is also a boxed type, it can be used for signal marshallers and GObject properties at zero cost.

This whole thing seems pretty efficient, and fairly simple to wrap your head around, but things fall apart pretty quickly as soon as you make this API public and tell people to use it from languages that are not C.

As I said above, boxed types cannot have sub-types; the type system has no idea that DerivedA implements the Base API contract. Additionally, since the whole introspection system is based on conventions applied on top of some C API, there is no way for language bindings to know that the derived_a_get_some_other_field() function is really a DerivedA method, meant to operate on DerivedA instances. Instead, you’ll only be able to access the method as a static function, like:

obj = Namespace.derived_a_new()
Namespace.derived_a_get_some_other_field(obj)

instead of the idiomatic, and natural:

obj = Namespace.DerivedA.new()
obj.get_some_other_field()

In short: please, don’t use boxed types for this, unless you’re planning to hide this functionality from the public API.

Typed instances

At this point the recommendation would be to switch to GObject for your type; make the type derivable in your project’s scope, avoid properties and signals, and you get fairly idiomatic code, and a bunch of other features, like weak references, toggle references, and keyed instance data. You can use your types for properties and signals, and you’re pretty much done.

But what if you don’t want to use GObject…

Well, in that case GLib lets you create your own type hierarchy, with its own rules, by using GTypeInstance as the base type.

GTypeInstance is the common ancestor for everything that is meant to be derivable; it’s the base type for GObject as well. Implementing a GTypeInstance-derived hierarchy doesn’t take much effort: it’s mostly low level glue code:

instance-type.c[Lines 4-189]download

typedef struct _Base            Base;
typedef struct _BaseClass       BaseClass;
typedef struct _BaseTypeInfo    BaseTypeInfo;

#define BASE_TYPE               (base_get_type())
#define BASE_GET_CLASS(obj)     (G_TYPE_INSTANCE_GET_CLASS ((obj), BASE_TYPE, 

// Simple macro that lets you chain up to the parent type's implementation
// of a virtual function, e.g.:
//
//   BASE_SUPER (self)->finalize (obj);
#define BASE_SUPER(obj)         ((BaseClass *) g_type_class_peek (g_type_parent (G_TYPE_FROM_INSTANCE (obj))))

struct _BaseClass
{
  GTypeClass parent_class;

  void  (* finalize)    (Base *self);
  void  (* foo)         (Base *self);
  void  (* bar)         (Base *self);
}

struct _Base
{
  GTypeInstance parent_instance;

  gatomicrefcount ref_count;

  // Shared field
  int some_field;
};

// A structure to be filled out by derived types when registering
// themselves into the type system; it copies the vtable into the
// class structure, and defines the size of the instance
struct _BaseTypeInfo
{
  gsize instance_size;

  void  (* finalize)    (Base *self);
  void  (* foo)         (Base *self);
  void  (* bar)         (Base *self);
};

// GValue table, so that you can initialize, compare, and clear
// your type inside a GValue, as well as collect/copy it when
// dealing with variadic arguments
static void
value_base_init (GValue *value)
{
  value->data[0].v_pointer = NULL;
}

static void
value_base_free_value (GValue *value)
{
  if (value->data[0].v_pointer != NULL)
    base_unref (value->data[0].v_pointer);
}

static void
value_base_copy_value (const GValue *src,
                       GValue       *dst)
{
  if (src->data[0].v_pointer != NULL)
    dst->data[0].v_pointer = base_ref (src->data[0].v_pointer);
  else
    dst->data[0].v_pointer = NULL;
}

static gpointer
value_expression_peek_pointer (const GValue *value)
{
  return value->data[0].v_pointer;
}

static char *
value_base_collect_value (GValue      *value,
                          guint        n_collect_values,
                          GTypeCValue *collect_values,
                          guint        collect_flags)
{
  Base *base = collect_values[0].v_pointer;

  if (base == NULL)
    {
      value->data[0].v_pointer = NULL;
      return NULL;
    }

  if (base->parent_instance.g_class == NULL)
    return g_strconcat ("invalid unclassed Base pointer for "
                        "value type '",
                        G_VALUE_TYPE_NAME (value),
                        "'",
                        NULL);

  value->data[0].v_pointer = base_ref (base);

  return NULL;
}

static gchar *
value_base_lcopy_value (const GValue *value,
                        guint         n_collect_values,
                        GTypeCValue  *collect_values,
                        guint         collect_flags)
{
  Base **base_p = collect_values[0].v_pointer;

  if (base_p == NULL)
    return g_strconcat ("value location for '",
                        G_VALUE_TYPE_NAME (value),
                        "' passed as NULL",
                        NULL);

  if (value->data[0].v_pointer == NULL)
    *base_p = NULL;
  else if (collect_flags & G_VALUE_NOCOPY_CONTENTS)
    *base_p = value->data[0].v_pointer;
  else
    *base_p = base_ref (value->data[0].v_pointer);

  return NULL;
}

// Register the Base type
GType
base_get_type (void)
{
  static volatile gsize base_type__volatile;

  if (g_once_init_enter (&base_type__volatile))
    {
      // This is a derivable type; we also want to allow
      // its derived types to be derivable
      static const GTypeFundamentalInfo finfo = {
        (G_TYPE_FLAG_CLASSED |
         G_TYPE_FLAG_INSTANTIATABLE |
         G_TYPE_FLAG_DERIVABLE |
         G_TYPE_FLAG_DEEP_DERIVABLE),
      };

      // The gunk for dealing with GValue
      static const GTypeValueTable value_table = {
        value_base_init,
        value_base_free_value,
        value_base_copy_value,
        value_base_peek_pointer,
        "p",
        value_base_collect_value,
        "p",
        value_base_lcopy_value,
      };

      // Base type information
      const GTypeInfo base_info = {
        // Class
        sizeof (GtkExpressionClass),
        (GBaseInitFunc) NULL,
        (GBaseFinalizeFunc) NULL,
        (GClassInitFunc) base_class_init,
        (GClassFinalizeFunc) NULL,
        NULL,

        // Instance
        sizeof (GtkExpression),
        0,
        (GInstanceInitFunc) base_init,

        // GValue
        &value_table,
      };

      // Register the Base type as a new, abstract fundamental type
      GType base_type =
        g_type_register_fundamental (g_type_fundamental_next (),
                                     g_intern_static_string ("Base"),
                                     &event_info, &finfo,
                                     G_TYPE_FLAG_ABSTRACT);

      g_once_init_leave (&base_type__volatile, expression_type);
    }

  return base_type__volatile;
}

Yes, this is a lot of code.

The base code stays pretty much the same:

instance-type.c[Lines 191-243]download

static void
base_real_finalize (Base *self)
{
  g_type_free_instance ((GTypeInstance *) self);
}

static void
base_class_init (BaseClass *klass)
{
  klass->finalize = base_real_finalize;
}

static void
base_init (Base *self)
{
  // Initialize the base instance fields
  g_atomic_ref_count_init (&res->ref_count);
  res->some_field = 42;
}

static Base *
base_alloc (GType type)
{
  g_assert (g_type_is_a (type, base_get_type());

  // Instantiate a new type derived by Base
  return (Base *) g_type_create_instance (type);
}

Base *
base_ref (Base *self)
{
  g_atomic_ref_count_inc (&self->ref_count);
}

void
base_unref (Base *self)
{
  if (g_atomic_ref_count_dec (&self->ref_count))
    BASE_GET_CLASS (self)->finalize (self);
}

void
base_foo (Base *self)
{
  BASE_GET_CLASS (self)->foo (self);
}

void
base_bar (Base *self)
{
  BASE_GET_CLASS (self)->bar (self);
}

except:

the reference counting is explicit, as we must use g_type_create_instance() and g_type_free_instance() to allocate and free the memory associated to the instance
you need to get the class structure from the instance using the GType macros instead of direct pointer access

Finally, you will need to add code to let you register derived types; since we want to tightly control the derivation, we use an ad hoc structure for the virtual functions, and we use a generic class initialization function:

instance-type.c[Lines 245-290]download

static void
base_generic_class_init (gpointer g_class,
                         gpointer class_data)
{
  BaseTypeInfo *info = class_data;
  BaseClass *klass = g_class;

  klass->finalize = info->finalize;
  klass->foo = info->foo;
  klass->bar = info->bar;

  // The info structure was copied, so we now need
  // to release the resources associated with it
  g_free (class_data);
}

// Register a derived typed of Base
static GType
base_type_register_static (const char         *type_name,
                           const BaseTypeInfo *type_info)
{
  // Simple check to ensure that the derived instance includes the
  // parent base type
  g_assert (type_info->instance_size >= sizeof (Base));

  GTypeInfo type_info;

  // All derived types have the same class and cannot add new virtual
  // functions
  type_info.class_size = sizeof (BaseClass);
  type_info.base_init = NULL;
  type_info.base_finalize = NULL;

  // Fill out the class vtable from the BaseTypeInfo structure
  type_info.class_init = base_generic_class_init;
  type_info.class_finalize = NULL;
  type_info.class_data = g_memdup (type_info, sizeof (BaseTypeInfo));

  // Instance information
  type_info.instance_size = type_info->instance_size;
  type_info.n_preallocs = 0;
  type_info.instance_init = NULL;
  type_info.value_table = NULL;

  return g_type_register_static (BASE_TYPE, type_name, &type_info, 0);
}

Otherwise, you could re-use the G_DEFINE_TYPE macro—yes, it does not require GObject—but then you’d have to implement your own class initialization and instance initialization functions.

After you defined the base type, you can structure your types in the same way as the boxed type code:

instance-type.c[Lines 294-347]download

typedef struct {
  Base parent;

  char *some_other_field;
} DerivedA;

static void
derived_a_finalize (Base *base)
{
  DerivedA *self = (DerivedA *) base;

  g_free (self->some_other_field);

  // We need to chain up to the parent's finalize() or we're
  // going to leak the instance
  BASE_SUPER (self)->finalize (base);
}

static const BaseTypeInfo derived_a_info = {
  .instance_size = sizeof (DerivedA),
  .finalize = derived_a_finalize,
  .foo = derived_a_foo, // defined elsewhere
  .bar = derived_a_bar, // defined elsewhere
};

GType
derived_a_get_type (void)
{
  static volatile gsize derived_type__volatile;

  if (g_once_init_enter (&derived_type__volatile))
    {
      // Register the type
      GType derived_type =
        base_type_register_static (g_intern_static_string ("DerivedA"),
                                   &derived_a_info);

      g_once_init_leave (&derived_type__volatile, derived_type);
    }

  return derived_type__volatile;
}

Base *
derived_a_new (const char *some_other_field)
{
  Base *res = base_alloc (derived_a_get_type ());

  DerivedA *self = (DerivedA *) res;

  self->some_other_field = g_strdup (some_other_field);

  return res;
}

The nice bit is that you can tell the introspection scanner how to deal with each derived type through annotations, and keep the API simple to use in C while idiomatic to use in other languages:

instance-type.c[Lines 349-363]download

/**
 * derived_a_get_some_other_field:
 * @base: (type DerivedA): a derived #Base instance
 *
 * Retrieves the `some_other_field` of a derived #Base instance.
 *
 * Returns: (transfer none): the contents of the field
 */
const char *
derived_a_get_some_other_field (Base *base)
{
  DerivedA *self = (DerivedA *) base;

  return self->some_other_field;
}

Cost-benefit

Of course, there are costs to this approach. In no particular order:

The type system boilerplate is a lot; the code size more than doubled from the boxed type approach. This is quite annoying, but at least it is a one-off cost, and you won’t likely ever need to change it. It would be nice to have it hidden by some magic macro incantation, but it’s understandably hard to do so without imposing restrictions on the kind of types you can create; since you’re trying to escape the restrictions of GObject, it would not make sense to impose a different set of restrictions.
If you want to be able to use this new type with properties and you cannot use G_TYPE_POINTER as a generic, hands-off container, you will need to derive GParamSpec, and add ad hoc API for GValue, which is even more annoying boilerplate. I’ve skipped it in the example, because that would add about 100 more lines of code.
Generated signal marshallers, and the generic one using libffi, do not know how to marshal typed instances; you will need custom written marshallers, or you’re going to use G_TYPE_POINTER everywhere and assume the risk of untyped boxing. The same applies to anything that uses the type system to perform things like serialization and deserialization, or GValue boxing and unboxing. You decided to build your own theme park on the Moon, and the type system has no idea how to represent it, or access its functionality.
Language bindings need to be able to deal with GTypeInstance and fundamental types; this is not always immediately necessary, so some maintainers do not add the code to handle this aspect of the type system.

The benefit is, of course, the fact that you are using a separate type hierarchy, and you get to make your own rules on things like memory management, lifetimes, and ownership. You can control the inheritance chain, and the rules on the overridable virtual functions. Since you control the whole type, you can add things like serialization and deserialization, or instance cloning, right at the top of the hierarchy. You could even implement properties without using GParamSpec.

Conclusion

Please, please use GObject. Writing type system code is already boring and error prone, which is why we added a ton of macros to avoid people shooting themselves in both their feet, and we hammered away all the special snowflake API flourishes that made parsing C API to generate introspection data impossible.

I can only recommend you go down the GTypeInstance route if you’ve done your due diligence on what that entails, and are aware that it is a last resort if GObject simply does not work within your project’s constraints.

Another layer

2019-08-15T10:16:00+01:00

Five years (and change) ago I was looking at the data types and API that were needed to write a 3D-capable scene graph API; I was also learning about SIMD instructions and compiler builtins on IA and ARM, as well as a bunch of math I didn’t really study in my brush offs with formal higher education. The result was a small library called Graphene.

Over the years I added more API, moved the build system from Autotools over to Meson, and wrote a whole separate library for its test suite.

In the meantime, GStreamer started using Graphene in its GL element; GTK 3.9x is very much using Graphene internally and exposing it as public API; Mutter developers are working on reimplementing the various math types in their copies of Cogl and Clutter using Graphene; and Alex wrote an entire 3D engine using it.

Not bad for a side project.

Of course, now I’ll have to start maintaining Graphene like a proper grownup, which also means reporting its changes, bug fixes, and features when I’m at the end of a development cycle.

While the 1.8 development cycle consisted mostly of bug fixes with no new API, there have been a few major internal changes during the development cycle towards 1.10:

I rewrote the Euler angles conversion to and from quaternions and matrices; the original implementation I cribbed from here and there was not really adequate, and broke pretty horribly when you tried to roundtrip from Euler angles to a transformation matrix and back. This also affected the conversion between Euler angles and quaternions. The new implementation is more correct, and as a side effect it now includes not just the Tait–Bryan angles, but also the classic Euler angles. All possible orders are available in both the intrinsic and extrinsic axes variants.
We’re dealing with floating point comparison and with infinities a bit better, now; this is usually necessary because the various vector implementations may have different behaviour, depending on the toolchain in use. A shout out goes to Intel, who bothered to add an instruction to check for infinities only in AVX 512, making it pointless for me, and causing a lot more grief than necessary.
The ARM NEON implementation of graphene_simd4f_t has been fixed and tested on actual ARM devices (an old Odroid I had lying around for ARMv7 and a Raspberry Pi3 for Aarch64); this means that the “this is experimental” compiler warning has been removed. I still need to run the CI on an ARM builder, but at least I can check if I’m doing something dumb, now.
As mentioned in the blog posts above, the whole test suite has been rewritten using µTest, which dropped a dependency on GLib; you still need GLib to get the integration with GObject, but if you’re not using that, Graphene should now be easier to build and test.

On the API side:

there are a bunch of new functions for graphene_rect_t, courtesy of Georges Basile Stavracas Neto and Marco Trevisan
thanks to Marco, the graphene_rect_round() function has been deprecated in favour of the more reliable graphene_rect_round_extents()
graphene_quaternion_t gained new operators, like add(), multiply() and scale()
thanks to Alex Larsson, graphene_plane_t can now be transformed using a matrix
I added equality and near-equality operators for graphene_matrix_t, and a getter function to retrieve the translation components of a transformation matrix
I added interpolation functions for the 2, 3, and 4-sized vectors
I’m working on exposing the matrix decomposition code for Gthree, but that requires some untangling of messy code so I’ll be in the next development snapshot.

On the documentation side:

I’ve reworked the contribution guide, and added a code of conduct to the project; doesn’t matter how many times you say “patches welcome” if you also aren’t clear on how those patches should be written, submitted, and reviewed, and if you aren’t clear on what constitutes acceptable behaviour when it comes to interactions between contributors and the maintainer
this landed at the tail end of 1.8, but I’ve hopefully clearly documented the conventions of the matrix/matrix and matrix/vector operations, to the point that people can use the Graphene API without necessarily having to read the code to understand how to use it

This concludes the changes that will appear with the next 1.10 stable release, which will be available by the time GNOME 3.34 is out. For the time being, you can check out the latest development snapshot available on Github.

I don’t have many plans for the future, to be quite honest; I’ll keep an eye out for what GTK and Gthree need, and I expect that once Mutter starts using Graphene I’ll start receiving bug reports.

One thing I did try was moving to a “static” API reference using Markdeep, just like I did for µTest, and drop yet another dependency; sadly, since we need to use gtk-doc annotations for the GObject introspection data generation, we’re going to depend on gtk-doc for a little while longer.

Of course, if you are using Graphene and you find some missing functionality, feel free to open an issue, or a merge request.

More little testing

2019-06-01T12:15:00+01:00

Back in March, I wrote about µTest, a Behavior-Driven Development testing API for C libraries, and that I was planning to use it to replace the GLib testing API in Graphene.

As I was busy with other things in GTK, it took me a while to get back to µTest—especially because I needed some time to set up a development environment on Windows in order to port µTest there. I managed to find some time over various weekends and evenings, and ended up fixing a couple of small issues here and there, to the point that I could run µTest’s own test suite on my Windows 10 box, and then get the CI build job I have on Appveyor to succeed as well.

Setting up MSYS2 was the most time consuming bit, really

While at it, I also cleaned up the API and properly documented it.

Since depending on gtk-doc would defeat the purpose, and since I honestly dislike Doxygen, I was looking for a way to write the API reference and publish it as HTML. As luck would have it, I remembered a mention on Twitter about Markdeep, a self-contained bit of JavaScript capable of turning a Markdown document into a half decent HTML page client side. Coupled with GitHub pages, I ended up with a fairly decent online API reference that also works offline, falls back to a Markdown document when not running through JavaScript, and can get fixed via pull requests.

Now that µTest is in a decent state, I ported the Graphene test suite over to it and, now I can run it on Windows using MSVC—and MSYS2, as soon as the issue with GCC gets fixed upstream. This means that, hopefully, we won’t have regressions on Windows in the future.

The µTest API is small enough, now, that I don’t plan major changes; I don’t want to commit to full API stability just yet, but I think we’re getting close to a first stable release soon; definitely before Graphene 1.10 gets released.

In case you think this could be useful for you: feedback, in the form of issues and pull requests, is welcome.

(New) Adventures in CI

2019-04-13T17:33:00+01:00

One of the great advantages of moving the code hosting in GNOME to GitLab is the ability to run per-project, per-branch, and per-merge request continuous integration pipelines. While we’ve had a CI pipeline for the whole of GNOME since 2012, it is limited to the master branch of everything, so it only helps catching build issues post-merge. Additionally, we haven’t been able to run test suites on Continuous since early 2016.

Being able to run your test suite is, of course, great—assuming you do have a test suite, and you’re good at keeping it working; gating all merge requests on whether your CI pipeline passes or fails is incredibly powerful, as it not only keeps your from unknowingly merging broken code, but it also nudges you in the direction of never pushing commits to the master branch. The downside is that it lacks nuance; if your test suite is composed of hundreds of tests you need a way to know at a glance which ones failed. Going through the job log is kind of crude, and it’s easy to miss things.

Luckily for us, GitLab has the ability to create a cover report for your test suite results, and present it on the merge request summary, if you generate an XML report and tell the CI machinery where you put it:

  artifacts:
    reports:
      junit:
        - "${CI_PROJECT_DIR}/_build/report.xml"

Sadly, the XML format chosen by GitLab is the one generated by JUnit, and we aren’t really writing Java classes. The JUnit XML format is woefully underdocumented, with only an unofficial breakdown of the entities and structure available. On top of that, since JUnit’s XML format is undocumented, GitLab has its own quirks in how it parses it.

Okay, assuming we have nailed down the output, how about the input? Since we’re using Meson on various projects, we can rely on machine parseable logs for the test suite log. Unfortunately, Meson currently outputs something that is not really valid JSON—you have to break the log into separate lines, and parse each line into a JSON object, which is somewhat less than optimal. Hopefully future versions of Meson will generate an actual JSON file, and reduce the overhead in the tooling consuming Meson files.

Nevertheless, after an afternoon of figuring out Meson’s output, and reverse engineering the JUnit XML format and the GitLab JUnit parser, I managed to write a simple script that translates Meson’s testlog.json file into a JUnit XML report that you can use with GitLab after you ran the test suite in your CI pipeline. For instance, this is what GTK does:

set +e

xvfb-run -a -s "-screen 0 1024x768x24" \
    meson test \
         -C _build \
         --timeout-multiplier 2 \
     --print-errorlogs \
     --suite=gtk \
     --no-suite=gtk:gsk \
     --no-suite=gtk:a11y

# Save the exit code, so we can reuse it
# later to pass/fail the job
exit_code=$?

# We always run the report generator, even
# if the tests failed
$srcdir/.gitlab-ci/meson-junit-report.py \
        --project-name=gtk \
        --job-id="${CI_JOB_NAME}" \
        --output=_build/${CI_JOB_NAME}-report.xml \
        _build/meson-logs/testlog.json

exit $exit_code

Which results in this:

Some assembly required; those are XFAIL reftests, but JUnit doesn’t understand the concept

The JUnit cover report in GitLab is only shown inside the merge request summary, so it’s not entirely useful if you’re developing in a branch without opening an MR immediately after you push to the repository. I prefer working on feature branches and getting the CI to run on my changes without necessarily having to care about opening the MR until my work is ready for review—especially since GitLab is not a speed demon when it comes to MRs with lots of rebases/fixup commits in them. Having a summary of the test suite results in that case is still useful, so I wrote a small conversion script that takes the testlog.json and turns it into an HTML page, with a bit of Jinja templating thrown into it to avoid hardcoding the whole thing into string chunks. Like the JUnit generator above, we can call the HTML generator right after running the test suite:

$srcdir/.gitlab-ci/meson-html-report.py \
        --project-name=GTK \
        --job-id="${CI_JOB_NAME}" \
        --output=_build/${CI_JOB_NAME}-report.html \
        _build/meson-logs/testlog.json

Then, we take the HTML file and store it as an artifact:

  artifacts:
    when: always
    paths:
      - "${CI_PROJECT_DIR}/_build/${CI_JOB_NAME}-report.html"

And GitLab will store it for us, so that we can download it or view it in the web UI.

There are additional improvements that can be made. For instance, the reftests test suite in GTK generates images, and we’re already uploading them as artifacts; since the image names are stable and determined by the test name, we can create a link to them in the HTML report itself, so we can show the result of the failed tests. With some more fancy HTML, CSS, and JavaScript, we could have a nicer output, with collapsible sections hiding the full console log. If we had a place to upload test results from multiple pipelines, we could even graph the trends in the test suite on a particular branch, and track our improvements.

All of this is, of course, not incredibly novel; nevertheless, the network effect of having a build system in Meson that lends itself to integration with additional tooling, and a code hosting infrastructure with native CI capabilities in GitLab, allows us to achieve really cool results with minimal glue code.

A little testing

2019-03-14T15:01:00+00:00

Years ago I started writing Graphene as a small library of 3D transformation-related math types to be used by GTK (and possibly Clutter, even if that didn’t pan out until Georges started working on the Clutter fork inside Mutter).

Graphene’s only requirement is a C99 compiler and a decent toolchain capable of either taking SSE builtins or support vectorization on appropriately aligned types. This means that, unless you decide to enable the GObject types for each Graphene type, Graphene doesn’t really need GLib types or API—except that’s a bit of a lie.

As I wanted to test what I was doing, Graphene has an optional build time dependency on GLib for its test suite; the library itself may not use anything from GLib, but if you want to build and run the test suite then you need to have GLib installed.

This build time dependency makes testing Graphene on Windows a lot more complicated than it ought to be. For instance, I need to install a ton of packages when using the MSYS2 toolchain on the CI instance on AppVeyor, which takes roughly 6 minutes each for the 32bit and the 64bit builds; and I can’t build the test suite at all when using MSVC, because then I’d have to download and build GLib as well—and just to access the GTest API, which I don’t even like.

What’s wrong with GTest

GTest is kind of problematic—outside of Google hijacking the name of the API for their own testing framework, which makes looking for it a pain. GTest is a lot more complicated than a small unit testing API needs to be, for starters; it was originally written to be used with a specific harness, gtester, in order to generate a very brief HTML report using gtester-report, including some timing information on each unit—except that gtester is now deprecated because the build system gunk to make it work was terrible to deal with. So, we pretty much told everyone to stop bothering, add a --tap argument when calling every test binary, and use the TAP harness in Autotools.

Of course, this means that the testing framework now has a completely useless output format, and with it, a bunch of default behaviours driven by said useless output format, and we’re still deciding if we should break backward compatibility to ensure that the supported output format has a sane default behaviour.

On top of that, GTest piggybacks on GLib’s own assertion mechanism, which has two major downsides:

it can be disabled at compile time by defining G_DISABLE_ASSERT before including glib.h, which, surprise, people tend to use when releasing; thus, you can’t run tests on builds that would most benefit from a test suite
it literally abort()s the test unit, which breaks any test harness in existence that does not expect things to SIGABRT midway through a test suite—which includes GLib’s own deprecated gtester harness

To solve the first problem we added a lot of wrappers around g_assert(), like g_assert_true() and g_assert_no_error(), that won’t be disabled depending on your build options and thus won’t break your test suite—and if your test suite is still using g_assert(), you’re strongly encouraged to port to the newer API. The second issue is still standing, and makes running GTest-based test suite under any harness a pain, but especially under a TAP harness, which requires listing the amount of tests you’ve run, or that you’re planning to run.

The remaining issues of GTest are the convoluted way to add tests using a unique path; the bizarre pattern matching API for warnings and errors; the whole sub-process API that relaunches the test binary and calls a single test unit in order to allow it to assert safely and capture its output. It’s very much the GLib test suite, except when it tries to use non-GLib API internally, like the command line option parser, or its own logging primitives; it’s also sorely lacking in the GObject/GIO side of things, so you can’t use standard API to create a mock GObject type, or a mock GFile.

If you want to contribute to GLib, then working on improving the GTest API would be a good investment of your time; since my project does not depend on GLib, though, I had the chance of starting with a clean slate.

A clean slate

For the last couple of years I’ve been playing off and on with a small test framework API, mostly inspired by BDD frameworks like Mocha and Jasmine. Behaviour Driven Development is kind of a buzzword, like test driven development, but I particularly like the idea of describing a test suite in terms of specifications and expectations: you specify what a piece of code does, and you match results to your expectations.

The API for describing the test suites is modelled on natural language (assuming your language is English, sadly):

  describe("your data type", function() {
    it("does something", () => {
      expect(doSomething()).toBe(true);
    });
    it("can greet you", () => {
      let greeting = getHelloWorld();
      expect(greeting).not.toBe("Goodbye World");
    });
  });

Of course, C is more verbose that JavaScript, but we can adopt a similar mechanism:

static void
something (void)
{
  expect ("doSomething",
    bool_value (do_something ()),
    to_be, true,
    NULL);
}

static void
{
  const char *greeting = get_hello_world ();

  expect ("getHelloWorld",
    string_value (greeting),
    not, to_be, "Goodbye World",
    NULL);
}

static void
type_suite (void)
{
  it ("does something", do_something);
  it ("can greet you", greet);
}

…
  describe ("your data type", type_suite);
…

If only C11 got blocks from Clang, this would look a lot less clunkier.

The value wrappers are also necessary, because C is only type safe as long as every type you have is an integer.

Since we’re good C citizens, we should namespace the API, which requires naming this library—let’s call it µTest, in a fit of unoriginality.

One of the nice bits of Mocha and Jasmine is the output of running a test suite:

$ ./tests/general 

  General
    contains at least a spec with an expectation
      ✓ a is true
      ✓ a is not false

      2 passing (219.00 µs)

    can contain multiple specs
      ✓ str contains 'hello'
      ✓ str contains 'world'
      ✓ contains all fragments

      3 passing (145.00 µs)

    should be skipped
      - skip this test

      0 passing (31.00 µs)
      1 skipped


Total
5 passing (810.00 µs)
1 skipped

Or, with colors:

Using colors means immediately taking this more seriously

The colours go automatically away if you redirect the output to something that is not a TTY, so your logs won’t be messed up by escape sequences.

If you have a test harness, then you can use the MUTEST_OUTPUT environment variable to control the output; for instance, if you’re using TAP you’ll get:

$ MUTEST_OUTPUT=tap ./tests/general
# General
# contains at least a spec with an expectation
ok 1 a is true
ok 2 a is not false
# can contain multiple specs
ok 3 str contains 'hello'
ok 4 str contains 'world'
ok 5 contains all fragments
# should be skipped
ok 6 # skip: skip this test
1..6

Which can be passed through to prove to get:

$ MUTEST_OUTPUT=tap prove ./tests/general
./tests/general .. ok
All tests successful.
Files=1, Tests=6,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
Result: PASS

I’m planning to add some additional output formatters, like JSON and XML.

Using µTest

Ideally, µTest should be used as a sub-module or a Meson sub-project of your own; if you’re using it as a sub-project, you can tell Meson to build a static library that won’t get installed on your system, e.g.:

mutest_dep = dependency('mutest-1',
  fallback: [ 'mutest', 'mutest_dep' ],
  default_options: ['static=true'],
  required: false,
  disabler: true,
)

# Or, if you're using Meson < 0.49.0
mutest_dep = dependency('mutest-1', required: false)
if not mutest_dep.found()
  mutest = subproject('mutest',
    default_options: [ 'static=true', ],
    required: false,
  )

  if mutest.found()
    mutest_dep = mutest.get_variable('mutest_dep')
  else
    mutest_dep = disabler()
  endif
endif

Then you can make the tests conditional on mutest_dep.found().

µTest is kind of experimental, and I’m still breaking its API in places, as a result of documenting it and trying it out, by porting the Graphene test suite to it. There’s still a bunch of API that I’d like to land, like custom matchers/formatters for complex data types, and a decent want to skip a specification or a whole suite; plus, as I said above, some additional formatted output.

If you have feedback, feel free to open an issue—or a pull request wink wink nudge nudge.

And I’m home

2018-12-14T19:00:00+00:00

It’s almost the end of the year, so it’s time for a recap of the previous episodes, I guess.

The tl;dr of 2018: started pretty much the same; massive dip in the middle; and, finally, got better at the very end.

The first couple of months of the year were pretty good; had a good time at the GTK hackfest and FOSDEM, and went to the Recipes hackfest in Yogyakarta in February.

In March, my wife Marta was diagnosed with breast cancer; Marta already had (different types of) cancer twice in her life, and had been in full remission for a couple of years, which meant she was able to cope with the mechanics of the process, but it was still a solid blow. Since she had already gone through a round of radiotheraphy 20 years ago—which had likely a hand in the cancer appearing now—her only option was surgery to remove the whole breast tissue and the associated lymph nodes. Not fun, but surgery went well, and she didn’t even need chemotherapy, so all in all it could have been way, way worse.

While Marta and I were dealing with that, I suddenly found myself out of a job, after working five years at Endless.

To be fair, this left me with enough time to help out Marta while she was recovering—which is why I didn’t come to GUADEC. After Marta was back on her feet, and was able to raise her right arm above her head, I took the first vacation in, I think, about four years. I relaxed, read a bunch of books, played some video games, built many, many, many Gundam plastic models, recharged my batteries—and ended up finally having time to spend on a project that I had pushed back for a while, because I really needed to add writing and producing 15 to 20 minutes of audio every week, after perusing thousands of email archives and old web pages on the Wayback Machine. Side note: donate to the Wayback Machine, if you can. They provide a fundamental service for everybody using the Web, and especially for people like me who want to trace the history of things that happen on the Web.

Of course I couldn’t stay home playing video games, recording podcasts, and building gunplas forever, and so I had to figure out where to go to work next, as I do enjoy being able to have a roof above my head, as well as buying food and stuff. By a crazy random happenstance, the GNOME Foundation announced that, thanks to a generous anonymous donation, it would start hiring staff, and that one of the open positions was for a GTK developer. I decided to apply, as, let’s be honest, it’s basically the dream job for me. I’ve been contributing to GNOME components for about 15 years, and to GTK for 12; and while I’ve been paid to contribute to some GNOME-related projects over the years, it was always as part of non-GNOME related work.

The hiring process was really thorough, but in the end I managed to land the most amazing job I could possibly hope for.

If you’re wondering what I’ll be working on, here’s a rough list:

improving performance, especially on less powerful devices
identify and land new features
identify and fix pain points for current consumers of GTK

On top of that, I’ll try to do my best to increase the awareness of the work being done on both the GTK 3.x stable branch, and the 4.x development branch, so expect more content appearing on the development blog.

The overall idea is to ensure that GTK gets more exposure and mindshare in the next 5 years as the main toolkit for Linux and Unix-like operating systems, as well better functionality for application developers that want to make sure their projects work on other platforms.

Finally, we want to make sure that more people feel confident enough to contribute to the core application development platform; if you have your pet feature or your pet bug inside GTK, and you want guidance, feel free to reach out to me.

Hopefully, the next year will not look like this one, and will be a bit better. Of course, if we in the UK don’t all die in the fiery chaos that is the Brexit circus…

The History of GNOME

2018-10-25T19:00:00+01:00

I’ve done a thing which may be of interest if you’re following the GNOME community.

As I said on Twitter, I have spare time, and I like boring people to death by talking about things that matter to me a lot; one of the things that matter to me is GNOME and its community—and especially its history.

Of course, I had to go and make it about liminal spaces and magic rituals, because that’s what makes it fun. This, though, is a magic ritual. I’m holding a seance, and I’m calling forth the past of the GNOME project for the people that live down its light-cone.

GNOME has the luxury of having a lot of people that stuck around—some even from the early days when there was no GNOME; there are also other people, though, some of them born after Miguel’s announcement, that are now starting to contribute to GNOME. I guess that means that it’s time to look back a bit, and give some more context to the history of the project.

I hope I won’t bore you that much with this; I hope that people will learn something new, or re-discover something that was forgotten. In general, I do hope people will have fun with it.

Reference counting in modern GLib

2018-09-04T16:35:00+01:00

Reference counting is a fairly common garbage collection technique used in many projects; the core GNOME platform uses pretty much all the time, from container data types, to GObject.

Implementing reference counting in C is typically fairly easy. Add a field to your data type:

typedef struct {
  int ref_count;

  char *name;
  char *address;
  char *city;

  int age;
} Person;

Then initialise it when creating a new instance:

Person *
person_new (void)
{
  Person *res = g_new0 (Person, 1);

  res->ref_count = 1;

  return res;
}

Instead of a person_copy() and a person_free() pair of functions, you need to write a person_ref() and a person_unref() pair:

Person *
person_ref (Person *p)
{
  // Acquire a reference
  p->ref_count++;

  return p;
}

static void
person_free (Person *p)
{
  // Free the data
  g_free (p->name);
  g_free (p->address);
  g_free (p->city);

  // Free the instance
  g_free (p);
}

void
person_unref (Person *p)
{
  // Release a reference
  p->ref_count--;

  // If this was the last reference, free the data
  // associated to the instance and then the instance
  // itself
  if (p->ref_count == 0)
    {
      person_free (p);
    }
}

Of course, trivial doesn’t mean correct. For instance, the code above assumes that all reference acquisition and release operations will happen from the same thread; if that’s not the case, you’ll have to use atomic integer operations to increase and decrease the reference count. Additionally, the code above does not check for overflows in the reference counting, which means that the value could saturate and lead to leaks.

Reimplementing all the checks and behaviours is not only boring, but it inevitably leads to mistakes along the way. For this reason, GLib 2.58 introduced two new types:

grefcount, to implement simple reference counting
gatomicrefcount, to implement atomic reference counting

Both come with their own API, which leads to this code:

typedef struct {
  grefcount ref_count;
  // or: gatomicrefcount ref_count;

  // same as above
} Person;

Person *
person_new (void)
{
  Person *res = g_new0 (Person, 1);

  g_ref_count_init (&res->ref_count);
  // or: g_atomic_ref_count_init (&res->ref_count);

  return res;
}

Person *
person_ref (Person *p)
{
  g_ref_count_inc (&p->ref_count);
  // or: g_atomic_ref_count_inc (&p->ref_count);

  return p;
}

void
person_unref (Person *p)
{
  if (g_ref_count_dec (&p->ref_count))
  // or: if (g_atomic_ref_count_dec (&p->ref_count))
    {
      person_free (p);
    }
}

The grefcount and gatomicrefcount make it immediately obvious that the field is used to implement reference counting semantics; the API checks for saturation of the counters, and will emit a warning; the atomic operations are verified. Additionally, the API checks if you’re trying to pass an atomic reference count to the grefcount API, and vice versa, so you have a layer of protection there during eventual refactoring, even if the types are both integer aliases.

We did not stop there, though.

Adding a reference count field to a structure is not always possible; for instance, if you have ABI compatibility restrictions, or if the type defition is public, adding a field may just not be something you can do within the same API version. By way of an example: you may have a type meant to be typically placed on the stack, so it needs a public, complete declaration, in order to have a well-defined size at compile time. You may also want to pass around an instance of the type as the argument for a GObject signal, or as a property — but it may be expensive to copy the data around, so you really want to have an optional reference counting mechanism that is invisible to the vast majority of the use cases.

This is why we also added an allocator function that adds reference counting semantics to the memory areas it returns, called GRcBox:

typedef struct {
  char *name;
  char *address;
  char *city;

  int age;
} Person;

Person *
person_new (void)
{
  return g_rc_box_new0 (Person);
}

Person *
person_ref (Person *p)
{
  return g_rc_box_acquire (p);
}

void
person_unref (Person *p)
{
  // person_free() is copied from the code above; we use
  // g_rc_box_release_full() because we have data to free
  // as well, but there's a variant for structures that
  // do not have internal allocations
  g_rc_box_release_full (p, person_free);
}

As you can see, this cuts down the boilerplate and repetition considerably.

The data returned to you by g_rc_box_new() and friends is exactly the same as you’d get from g_new(), so you can pass it around to your API exactly the same — but it is transparently augmented with reference counting semantics. You acquire and release references without having an explicit reference counter. The only restriction is that you cannot reallocate the data, so calling realloc() is not allowed; and, of course, you cannot free the memory directly with free() — you need to release the last reference.

Similar to GRcBox, there’s a GAtomicRcBox, which provides atomic reference counting semantics, with a similar API.

Both GRcBox and GAtomicRcBox are Valgrind-safe, so you won’t get false positives or unreachable memory if run your code under Valgrind.

You don’t even need to have a structure to allocate: you can use GRcBox to allocate any memory area with a non-zero size. Incidentally, this is how we implemented a oft-requested feature: reference counted strings, which are also available in GLib 2.58.

Reference counted strings work exactly like any other C string, but instead of copying them and freeing them, you acquire and release references to them:

char *s = g_ref_string_new ("hello, world!");

// "s" is just like any other C string
g_print ("s['%s'] length = %d\n", s, strlen (s));

g_ref_string_release (s);

Reference counted strings can also be interned, which means that calling g_ref_string_new_intern() with the same string twice will give you a new reference the second time around, instead of allocating a new string; once the last reference to the interned string is dropped, the string is un-interned, and the resource allocated for it are freed.

Since you may be wary of passing around char * for reference counted strings, there’s a handy GRefString C type you can use to improve readability, like we have GStrv for char **:

GRefString *s = g_ref_string_new ("hello");

// ...

g_ref_string_release (s);

GRefString also has an autocleanup function, so you can do:

{
  g_autoptr(GRefString) s = g_ref_string_new ("hello!");

  // ...
}

and the string will automatically be released once it goes out of scope.

pkg-config and paths

2018-03-15T16:45:00+00:00

This is something of a frequently asked question, as it comes up every once in a while. The pkg-config documentation is fairly terse, and even pkgconf hasn’t improved on that.

The problem

Let’s assume you maintain a project that has a dependency using pkg-config.

Let’s also assume that the project you are depending on loads some files from a system path, and your project plans to install some files under that path.

The questions are:

how can the project you are depending on provide an appropriate way for you to discover where that path is
how can the project you maintain use that information

The answer to both questions is: by using variables in the pkg-config file. Sadly, there’s still some confusion as to how those variables work, so this is my attempt at clarifying the issue.

Defining variables in pkg-config files

The typical preamble stanza of a pkg-config file is something like this:

prefix=/some/prefix
libdir=${prefix}/lib
datadir=${prefix}/share
includedir=${prefix}/include

Each variable can reference other variables; for instance, in the example above, all the other directories are relative to the prefix variable.

Those variables that can be extracted via pkg-config itself:

$ pkg-config --variable=includedir project-a
/some/prefix/include

As you can see, the --variable command line argument will automatically expand the ${prefix} token with the content of the prefix variable.

Of course, you can define any and all variables inside your own pkg-config file; for instance, this is the definition of the giomoduledir variable inside the gio-2.0 pkg-config file:

prefix=/usr
libdir=${prefix}/lib

…

giomoduledir=${libdir}/gio/modules

This way, the giomoduledir variable will be expanded to /usr/lib/gio/modules when asking for it.

If you are defining a path inside your project’s pkg-config file, always make sure you’re using a relative path!

We’re going to see why this is important in the next section.

Using variables from pkg-config files

Now, this is where things get complicated.

As I said above, pkg-config will expand the variables using the definitions coming from the pkg-config file; so, in the example above, getting the giomoduledir will use the prefix provided by the gio-2.0 pkg-config file, which is the prefix into which GIO was installed. This is all well and good if you just want to know where GIO installed its own modules, in the same way you want to know where its headers are installed, or where the library is located.

What happens, though, if your own project needs to install GIO modules in a shared location? More importantly, what happens if you’re building your project in a separate prefix?

If you’re thinking: “I should install it into the same location as specified by the GIO pkg-config file”, think again. What happens if you are building against the system’s GIO library? The prefix into which it has been installed is only going to be accessible by the administrator user; or it could be on a read-only volume, managed by libostree, so sudo won’t save you.

Since you’re using a separate prefix, you really want to install the files provided by your project under the prefix used to configure your project. That does require knowing all the possible paths used by your dependencies, hard coding them into your own project, and ensuring that they never change.

This is clearly not great, and it places additional burdens on your role as a maintainer.

The correct solution is to tell pkg-config to expand variables using your own values:

$ pkg-config \
> --define-variable=prefix=/your/prefix \
> --variable=giomoduledir
> gio-2.0
/your/prefix/lib/gio/modules

This lets you rely on the paths as defined by your dependencies, and does not attempt to install files in locations you don’t have access to.

Build systems

How does this work, in practice, when building your own software?

If you’re using Meson, you can use the get_pkgconfig_variable() method of the dependency object, making sure to replace variables:

gio_dep = dependency('gio-2.0')
giomoduledir = gio_dep.get_pkgconfig_variable(
  'giomoduledir',
  define_variable: [ 'libdir', get_option('libdir') ],
)

This is the equivalent of the --define-variable/--variable command line arguments.

If you are using Autotools, sadly, the PKG_CHECK_VAR m4 macro won’t be able to help you, because it does not allow you to expand variables. This means you’ll have to deal with it in the old fashioned way:

giomoduledir=`$PKG_CONFIG --define-variable=libdir=$libdir --variable=giomoduledir gio-2.0`

Which is annoying, and yet another reason why you should move off from Autotools and to Meson. 😃

Caveats

All of this, of course, works only if paths are expressed as locations relative to other variables. If that does not happen, you’re going to have a bad time. You’ll still get the variable as requested, but you won’t be able to make it relative to your prefix.

If you maintain a project with paths expressed as variables in your pkg-config file, check them now, and make them relative to existing variables, like prefix, libdir, or datadir.

If you’re using Meson to generate your pkg-config file, make sure that the paths are relative to other variables, and file bugs if they aren’t.

Recipes hackfest

2018-03-02T01:50:00+01:00

The Recipes application started as a celebration of GNOME’s community and history, and it’s grown to be a great showcase for what GNOME is about:

design guidelines and attention to detail
a software development platform for modern applications
new technologies, strongly integrated with the OS
people-centered development

Additionally, Recipes has become a place where to iterate design and technology for the rest of the GNOME applications.

Nevertheless, while design patterns, toolkit features, Flatpak and portals, are part of the development experience, without content provided by the people using Recipes there would not be an application to begin with.

If we look at the work Endless has been doing on its own framework for content-driven applications, there’s a natural fit — which is why I was really happy to attend the Recipes hackfest in Yogyakarta, this week.

Fried Jawanese noodle make a healty breakfast

In the Endless framework we take structured data — like a web page, or a PDF document, or a mix of video and text — and we construct “shards”, which embed both the content, its metadata, and a Xapian database that can be used for querying the data. We take the shards and distribute them though Flatpak as a runtime extension for our applications, which means we can take advantage of Flatpak for shipping updates efficiently.

During the hackfest we talked about how to take advantage of the data model Endless applications use, as well as its distribution model; instead of taking tarballs with the recipe text, the images, and the metadata attached to each, we can create shards that can be mapped to a custom data model. Additionally, we can generate those shards locally when exporting the recipes created by new chefs, and easily re-integrate them with the shared recipe shards — with the possibility, in the future, to have a whole web application that lets you submit new recipes, and the maintainers review them without necessarily going through Matthias’s email. 😉

The data model discussion segued into how to display that data. The Endless framework has the concept of cards, which are context-aware data views; depending on context, they can have more or less details exposed to the user — and all those details are populated from the data model itself. Recipes has custom widgets that do a very similar job, so we talked about how to create a shared layer that can be reused both by Endless applications and by GNOME applications.

Sadly, I don’t remember the name of this soup, only that it had chicken hearts in it, and that Cosimo loved it

At the end of the hackfest we were able to have a proof of concept of Recipes loading the data from a custom shard, and using the Endless framework to display it; translating that into shareable code and libraries that can be used by other projects is the next step of the roadmap.

All of this, of course, will benefit more than just the Recipes application. For instance, we could have a Dictionary application that worked offline, and used Wiktionary as a source, and allowed better queries than just substring matching; we could have applications like Photos and Documents reuse the same UI elements as Recipes for their collection views; Software and Recipes already share a similar “landing page” design (and widgets), which means that Software could also use the “card” UI elements.

There’s lots for everyone to do, but exciting times are ahead!

And after we’re done we can relax by the pool

I’d be remiss if I didn’t thank our hosts at the Amikom university.

Yogyakarta is a great city; I’ve never been in Indonesia before, and I’ve greatly enjoyed my time here. There’s lots to see, and I strongly recommend visiting. I’ve loved the food, and the people’s warmth.

I’d like to thank my employer, Endless, for letting me take some time to attend the hackfest; and the GNOME Foundation, for sponsoring my travel.

The travelling Wilber

GLib tools rewrite

2017-10-13T12:39:00+01:00

You can safely skip this article if you’re not building software using enumeration types and signal handlers; or if you’re already using Meson.

For more that 15 years, GLib has been shipping with two small utilities:

glib-mkenums, which scans a list of header files and generates GEnum and GFlags types out of them, for use in GObject properties and signals
glib-genmarshal, which reads a file containing a description of marshaller functions, and generates C code for you to use when declaring signals

If you update to GLib 2.54, released in September 2017, you may notice that the glib-mkenums and glib-genmarshal tools have become sligly more verbose and slightly more strict about their input.

During the 2.54 development cycle, both utilities have been rewritten in Python from a fairly ancient Perl, in the case of glib-mkenums; and from C, in the case of glib-genmarshal. This port was done to address the proliferation of build time dependencies on GLib; the cross-compilation hassle of having a small C utility being built and used during the build; and the move to Meson as the default (and hopefully only) build system for future versions of GLib. Plus, the port introduced colorised output, and we all know everything looks better with colors.

Sadly, none of the behaviours and expected input or output of both tools have ever been documented, specified, or tested in any way. Additionally, it turns out that lots of people either figured out how to exploit undefined behaviour, or simply cargo-culted the use of these tools into their own project. This is entirely on us, and I’m going to try and provide better documentation to both tools in the form of a decent man page, with examples of integration inside Autotools-based projects.

In the interest of keeping old projects building, both utilities will try to replicate the undefined behaviours as much as possible, but now you’ll get a warning instead of the silent treatment, and maybe you’ll get a chance at fixing your build.

If you are maintaining a project using those two utilities, these are the things to watch out for, and ideally to fix by strictly depending on GLib ≥ 2.54.

glib-genmarshal

if you’re using glib-genmarshal --header --body to avoid the “missing prototypes” compiler warning when compiling the generated marshallers source file, please switch to using --prototypes --body. This will ensure you’ll get only the prototypes in the source file, instead of a whole copy of the header.

Similarly, if you’re doing something like the stanza below in order to include the header inside the body:

foo-marshal.h: foo-marshal.list Makefile
        $(AM_V_GEN) \
          $(GLIB_GENMARSHAL) --header foo-marshal.list \
        > foo-marshal.h
foo-marshal.c: foo-marshal.h
        $(AM_V_GEN) (
          echo '#include "foo-marshal.h"' ; \
          $(GLIB_GENMARSHAL) --body foo-marshal.list \
        ) > foo-marshal.c

you can use the newly added --include-header command line argument, instead.

The stanza above has also been used to inject #define and #undef pre-processor directives; these can be replaced with the -D and -U newly added command line arguments, which work just like the GCC ones.
This is not something that came from the Python port, as it’s been true since the inclusion of glib-genmarshal in GLib, 17 years ago: the NONE and BOOL tokens are deprecated, and should not be used; use VOID and BOOLEAN, respectively. The new version of glib-genmarshal will now properly warn about this, instead of just silently converting them, and never letting you know you should fix your marshal.list file.

If you want to silence all messages outside of errors, you can now use the --quiet command line option; conversely, use --verbose if you want to get more messages.

glib-mkenums

The glib-mkenums port has been much more painful than the marshaller generator one; mostly, because there are many, many more ways to screw up code generation when you have command line options and file templates, and mostly because the original code base relied heavily on Perl behaviour and side effects. Cargo culting Autotools stanzas is also much more of a thing when it comes to enumerations than marshallers, apparently. Imagine what we could achieve if the tools that we use to build our code didn’t actively work against us.

First of all, try and avoid having mixed encoding inside source code files that are getting parsed; mixing Unicode and ISO-8859 encoding is not a great plan, and C does not have a way to specify the encoding to begin with. Yes, you may be doing that inside comments, so who cares? Well, a tool that parses comments might.
If you’re mixing template files with command line arguments for some poorly thought-out reason, like this:
```
foo-enums.h: foo-enums.h.in Makefile
        $(AM_V_GEN) $(GLIB_MKENUMS) \
          --fhead '#ifdef FOO_ENUMS_H' \
          --fhead '#defineFOO_ENUMS_H' \
          --template foo-enums.h.in \
          --ftail '#endif /* FOO_ENUMS_H */' \
        > foo-enums.h
```
the old version of glib-mkenums would basically build templates depending on the phase of the moon, as well as some internal detail of how Perl works. The new tool has a specified order:
- the HEAD stanzas specified on the command line are always prepended to the template file
- the PROD stanzas specified on the command line are always appended to the template file
- the TAIL stanzas specified on the command line are always appended to the template file

Like with glib-genmarshal, the glib-mkenums tool also tries to be more verbose in what it expects.

Ideally, by this point, you should have switched to Meson, and you’re now using a sane build system that generates this stuff for you.

If you’re still stuck with Autotools, though, you may also want to consider dropping glib-genmarshal, and use the FFI-based generic marshaller in your signal definitions — which comes at a small performance cost, but if you’re putting signal emission inside a performance-critical path you should just be ashamed of yourself.

For enumerations, you could use something like this macro, which I tend to employ in all my projects with just few, small enumeration types, and where involving a whole separate pass at parsing C files is kind of overkill. Ideally, GLib would ship its own version, so maybe it’ll be replaced in a new version.

Many thanks to Jussi Pakkanen, Nirbheek Chauhan, Tim-Philipp Müller, and Christoph Reiter for the work on porting glib-mkenums, as well as fixing my awful Parseltongue.

GUADEC 2017

2017-08-11T13:45:00+01:00

Another year, another GUADEC — my 13th to date. Definitely not getting younger, here. 😉

As usual, it was great to see so many faces, old and new. Lots of faces, as well; attendance has been really good, this year.

The 20th anniversary party was a blast; the venue was brilliant, and watching people going around the tables in order to fill in slots for the raffle tickets was hilarious. I loved every minute of it — even if the ‘90s music was an assault on my teenage years. See above, re: getting older.

Waiting for the #GnomeParty raffle to begin pic.twitter.com/BdCQDgV6UZ
— Emmanuele Bassi (@ebassi) July 29, 2017

The talks were, as usual, stellar. It’s always so hard to chose from the embarrassment of riches that is the submission pool, but every year I think the quality of what ends up on the schedule is so high that I cannot be sad.

First #guadec2017 keynote by @o0karen0o - “The Battle over our technology” pic.twitter.com/P0WnlASxe1
— Emmanuele Bassi (@ebassi) July 28, 2017

Flatpaks for the Flatpak throne! #guadec2017 pic.twitter.com/RrfLRdlgGN
— Emmanuele Bassi (@ebassi) July 28, 2017

Lots and lots of people were happy to see the Endless contingent at the conference; the talks from my colleagues were really well received, and I’m sure we’re going to see even more collaboration spring from the seeds planted this year.

My talk about continuous integration in GNOME was well-received, I think; I had to speed up a bit at the end because I lost time while connecting to the projector (not enough juice when on battery to power the HDMI-over-USB C connector; lesson learned for the next talk). I would have liked to get some more time to explain what I’d like to achieve with Continuous.

Do not disturb the build sheriff

I ended up talking with many people at the unconference days, in any case. If you’re interested in helping out the automated build of GNOME components and to improve the reliability of the project, feel free to drop by on irc.gnome.org (or on Matrix!) in the #testable channel.

The unconference days were also very productive, for me. The GTK+ session was, as usual, a great way to plan ahead for the future; last year we defined the new release cycle for GTK+ and jump start the 4.0 development cycle. This year we drafted a roadmap with the remaining tasks.

I talked about Flatpak, FlatHub, Builder, performance in Mutter and GNOME Shell; I wanted to attend the Rust and GJS sessions, but that would have required the ability to clone myself, or be in more than one place at once.

During the unconference, I was also able to finally finish the GDK-Pixbuf port of the build system to Meson. Testing is very much welcome, before we bin the Autotools build and bring one of the oldest modules in GNOME into the future.

Additionally, I was invited to the GNOME Release Team, mostly to deal with the various continuous integration build issues. This, sadly, does not mean that I’m one step closer to my ascendance as the power mad dictator of all of GNOME, but it means that if there are issues with your module, you have a more-or-less official point of contact.

I can’t wait for GUADEC 2018! See you all in Almería!

Dev v Ops

2017-08-10T16:51:00+01:00

In his talk at the 2017 GUADEC in Manchester, Richard Brown presented a set of objections to the current trend of new packaging systems — mostly AppImage, Snap, and Flatpak — from the perspective of a Linux distribution integrator.

I’m not entirely sure he managed to convince everybody in the attendance, but he definitely presented a well-reasoned argument, steeped in history. I freely admit I went in not expecting to be convinced, but fully expecting to be engaged and I can definitely say I left the room thoroughly satisfied, and full of questions on how we can make the application development and distribution story on Linux much better. Talking with other people involved with Flatpak and Flathub we already identified various places where things need to be improved, and how to set up automated tools to ensure we don’t regress.

In the end, though, all I could think of in order to summarise it when describing the presentation to people that did not attend it, was this:

Linux distribution developer tells application and system developers that packaging is a solved problem, as long as everyone uses the same OS, distribution, tools, and becomes a distro packager.

Which, I’m the first to admit, seems to subject the presentation to impressive levels of lossy compression. I want to reiterate that I think Richard’s argument was presented much better than this; even if the talk was really doom and gloom predictions from a person who sees new technologies encroaching in his domain, Richard had wholesome intentions, so I feel a bit bad about condensing them into a quip.

Of course, this leaves me in quite a bind. It would be easy — incredibly easy — to dismiss a lot of the objections and points raised by Richard as a case of the Italian idiom “do not ask to the inn-keeper if the house wine is good”. Nevertheless, I want to understand why those objections where made in the first place, because it’s not going to be the last we are going to be hearing them.

I’ve been turning an answer to that question in my head for a while, now, and I think I finally came up with something that tries to rise to the level of Richard’s presentation, in the sense that I tried to capture the issue behind it, instead of just reacting to it.

Like many things in tech, it all comes down to developers and system administators.

I don’t think I’m being controversial, or exposing some knowledge for initiates, when I say that Linux distributions are not made by the people that write the software they distribute. Of course, there are various exceptions, with upstream developers being involved (by volunteer or more likely paid work) with a particular distribution of their software, but by and large there has been a complete disconnect between who writes the code and who packages it.

Another, I hope, uncontroversial statement is that people on the Linux distribution side of things are mostly interested in making sure that the overall OS fits into a fairly specific view of how computer systems should work: a central authority that oversees, via policies and validation tools that implement those policies, how all the pieces fit together, up to a certain point. There’s a name for that kind of authority: system administrators.

Linux distributions are the continuation of system administration policies via other means: all installed Linux machines are viewed as part of the same shared domain, with clear lines of responsibility and ownership that trace from a user installation to the packager, to the people that set up the policies of integration, and which may or may not involve the people that make the software in the first place — after all, that’s what distro patches are for.

You may have noticed that in the past 35 years the landscape of computing has been changed by the introduction of the personal computer; that the release of Windows 95 introduced the concept of a mass marketable operating system; and that, by and large, there has been a complete disintermediation between software vendors and users. A typical computer user won’t have an administrator giving them a machine with the OS, validating and managing all the updates; instead of asking an admin to buy, test, and deploy an application for them, users went to a shop and bought a box with floppies or an optical storage — and now they just go to online version of that shop (typically owned by the OS vendor) and download it. The online store may just provide users with the guarantee that the app won’t steal all their money without asking in advance, but that’s pretty much where the responsibility of the owners of the store ends.

Linux does not have stores.

You’re still supposed to go ask your sysadmin for an application to be available, and you’re still supposed to give your application to the sysadmin so that they can deploy it — with or without modifications.

Yet, in the 25 years of their history, Linux distribution haven’t managed to convince the developers of

Perl
Python
Ruby
JavaScript
Rust
Go
PHP
insert_your_language_here

applications to defer all their dependency handling and integration to distro packagers.

They have just about managed to convince C and C++ developers, because the practices of those languages are so old and entrenched, the tools so poor, and because they share part of the same heritage; and TeX writers, for some weird reason, as you can witness by looking at how popular distribution package all the texlive modules.

The crux is that nobody, on any existing major (≥ 5% of market penetration) platform, develops applications like Linux distributors want them to. Nobody wants to. Not even the walled gardens you keep in your pocket and use to browse the web, watch a video, play a game, and occasionally make a phone call, work like that, and those are the closest thing to a heavily managed system you can get outside of a data center.

The issue is not the “managed by somebody” part; the issue is the inevitable intermediary between an application developer and an application user.

Application developers want to be able to test and have a reproducible environment, because it makes it easier for them to find bugs and to ensure that their project works as they intented; the easiest way to do that is to have people literally use the developer’s computer — this is why web applications deployed on top of a web browser engine that consumes all your CPU cores in a fiery pit are eating everybody’s lunch; or because software as a service even exists. The closest thing application developers have found to ship their working laptop to the users of our applications without physically shipping hardware, is to give them a read-only file system image that we have built ourselves, or a list of dependencies hosted on a public source code repository that the build system will automatically check out prior to deploying the application.

The Linux distribution model is to have system administrators turned packagers control all the dependencies and the way they interact on a system; check all the licensing terms and security issues, when not accidentally introducing them; and then fight among themselves on the practicalities and ideologies of how that software should be distributed, installed, and managed.

The more I think about it, the less I understand how that ever worked in the first place. It is not a mystery, though, why it’s a dying model.

When I say that “nobody develops applications like the Linux distributions encourages and prefers” I’m not kidding around: Windows, macOS, iOS, Electron, and Android application developers are heavily based on the concept of a core set of OS services; a parallel installable blocks of system dependencies shipped and retired by the OS vendor; and a bundling system that allows application developers to provide their own dependencies, and control them.

Sounds familiar?

If it does, it’s becase, in the past 25 years, every other platform (and I include programming languages with a fairly comprehensive standard library in that definition, not just operating systems) has implemented something like this — even in free and open source software, where this kind of invention mostly exists both as a way to replicate Linux distributions on Windows, and to route around Linux distributions on Linux.

It should not come as a surprise that there’s going to be friction; while for the past two decades architects of both operating systems and programming languages have been trying to come up with a car, Linux distributions have been investing immeasurable efforts in order to come up with a jet fueled, SRB-augmented horse. Sure: it’s so easy to run apt install foo and get foo installed. How did foo get into the repository? How can you host a repository, if you can’t, or don’t want to host it on somebody else’s infrastructure? What happens when you have to deal with a bajillion, slightly conflicting, ever changing policies? How do you keep your work up to date for everyone, and every combination? What happens if you cannot give out the keys to your application to everyone, even if the application itself may be free software?

Scalability is the problem; too many intermediaries, too many gatekeepers. Even if we had a single one, that’s still one too many. People using computers expect to access whatever application they need, at the touch of a finger or at the click of a pointer; if they cannot get to something in time for the task they have to complete, they will simply leave and never come back. Sure, they can probably appreciate the ease of installing 30 different console text editors, 25 IRC clients, and 12 email clients, all in various state of disrepair and functionality; it won’t really mean much, though, because they will be using something else by that time.

Of course, now we in the Linux world are in the situation of reimplementing the past 20 years of mistakes other platforms have made; of course, there will be growing pains, and maybe, if we’re careful enough, we can actually learn for somebody else’s blunders, and avoid falling into common traps. We’re going to have new and exciting traps to fall into!

Does this mean it’s futile, and that we should just give up on everything and just go back to our comfort zone? If we did, it would not only be a disservice to our existing users, but also to the users of every other platform. Our — and I mean the larger free software ecosystem — proposition is that we wish all users to have the tools to modify the software they are using; to ensure that the software in question has not been modified against their will or knowledge; and to access their own data, instead of merely providing it to third parties and renting out services with it. We should have fewer intermediaries, not more. We should push for adoption and access. We should provide a credible alternative to other platforms.

This will not be easy.

We will need to grow up a lot, and in little time; adopt better standards than just “it builds on my laptop” or “it works if you have been in the business for 15 years and know all the missing stairways, and by the way, isn’t that a massive bear trap covered with a tarpaulin on the way to the goal”. Complex upstream projects will have to start caring about things like reproducibility; licensing; security updates; continuous integration; QA and validation. We will need to care about stable system services, and backward compatibility. We will not be shielded by a third party any more.

The good news is: we have a lot of people that know about this stuff, and we can ask them how to make it work. We can take existing tools and make them generic and part of our build pipeline, instead of having them inside silos. We can adopt shared policies upstream instead of applying them downstream, and twisting software to adapt to all of them.

Again, this won’t be easy.

If we wanted easy, though, we would not be making free and open source software for everyone.

Further experiments in Meson

2017-05-19T18:20:00+01:00

Meson is definitely getting more traction in GNOME (and other projects), with many components adding support for it in parallel to autotools, or outright switching to it. There are still bugs, here and there, and we definitely need to improve build environments — like Continuous — to support Meson out of the box, but all in all I’m really happy about not having to deal with autotools any more, as well as being able to build the G* stack much more quickly when doing continuous integration.

Now that GTK+ has added Meson support, though, it’s time to go through the dependency chain in order to clean up and speed up the build in the lower bits of our stack. After an aborted attempt at porting GdkPixbuf, I decided to port Pango.

All in all, Pango proved to be an easy win; it took me about one day to port from Autotools to Meson, and most of it was mechanical translation from weird autoconf/automake incantations that should have been removed years ago¹. Most of the remaining bits were:

ensuring that both Autotools and Meson would build the same DSOs, with the same symbols
generating the same introspection data and documentation
installing tests and data in the appropriate locations

Thanks to the ever vigilant eye of Nirbheek Chauhan, and thanks to the new Meson reference, I was also able to make the Meson build slightly more idiomatic than a straight, 1:1 port would have done.

The results are a full Meson build that takes about the same time as ./autogen.sh to run:

* autogen.sh:                         * meson
  real        0m11.149s                 real          0m2.525s
  user        0m8.153s                  user          0m1.609s
  sys         0m2.363s                  sys           0m1.206s

* make -j$(($(nproc) + 2))            * ninja
  real        0m9.186s                  real          0m3.387s
  user        0m16.295s                 user          0m6.887s
  sys         0m5.337s                  sys           0m1.318s

--------------------------------------------------------------

* autotools                           * meson + ninja
  real        0m27.669s                 real          0m5.772s
  user        0m45.622s                 user          0m8.465s
  sys         0m10.698s                 sys           0m2.357s

Not bad for a day’s worth of work.

My plan would be to merge this in the master branch pretty soon; I also have a branch that drops Autotools entirely but that can wait a cycle, as far as I’m concerned.

Now comes the hard part: porting libraries like GdkPixbuf, ATK, gobject-introspection, and GLib to Meson. There’s already a GLib port, courtesy of Centricular, but it needs further testing; GdkPixbuf is pretty terrible, since it’s a really old library; I don’t expect ATK and GObject introspection to be complicated, but the latter has a non-recursive Make layout that is full of bees.

It would be nice to get to GUADEC and have the whole G* stack build with Meson and Ninja. If you want to help out, reach out in #gtk+, on IRC or on Matrix.

The Windows support still checks for GCC 2.x or 3.x flags, for instance. ↩

On Vala

2017-02-13T13:12:00+00:00

It seems I raised a bit of a stink on Twitter last week:

PSA: if you want to write a new @gnome application, don't use Vala; if you're already using it, consider porting to a non-dead language.
— Emmanuele Bassi (@ebassi) February 3, 2017

Of course, and with reason, I’ve been called out on this by various people. Luckily, it was on Twitter, so we haven’t seen articles on Slashdot and Phoronix and LWN with headlines like “GNOME developer says Vala is dead and will be removed from all servers for all eternity and you all suck”. At least, I’ve only seen a bunch of comments on Reddit about this, but nobody cares about that particular cesspool of humanity.

Sadly, 140 characters do not leave any room for nuance, so maybe I should probably clarify what I wrote on a venue with no character limit.

First of all, I’d like to apologise to people that felt I was attacking them or their technical choices: it was not my intention, but see above, re: character count. I may have only about 1000 followers on Twitter, but it seems that the network effect is still a bit greater than that, so I should be careful when wording opinions. I’d like to point out that it’s my private Twitter account, and you can only get to what it says if you follow me, or if you follow people who follow me and decide to retweet what I write.

My PSA was intended as a reflection on the state of Vala, and its impact on the GNOME ecosystem in terms of newcomers, from the perspective of a person that used Vala for his own personal projects; recommended Vala to newcomers; and has to deal with the various build issues that arise in GNOME because something broke in Vala or in projects using Vala. If you’re using Vala outside of GNOME, you have two options: either ignore all I’m saying, as it does not really apply to your case; or do a bit of soul searching, and see if what I wrote does indeed apply to you.

First of all, I’d like to qualify my assertion that Vala is a “dead language”. Of course people see activity in the Git repository, see the recent commits and think “the project is still alive”. Recent commits do not tell a complete story.

Let’s look at the project history for the past 10 cycles (roughly 2.5 years). These are the commits for every cycle, broken up in two values: one for the full repository, the other one for the whole repository except the vapi directory, which contains the VAPI files for language bindings:

Aside from the latest cycle, Vala has seen very little activity; the project itself, if we exclude binding updates, has seen less than 100 commits for every cycle — some times even far less. The latest cycle is a bit of an outlier, but we can notice a pattern of very little work for two/three cycles, followed by a spike. If we look at the currently in progress cycle, we can already see that the number of commits has decreased back to 55/42, as of this morning.

Number of commits is just a metric, though; more important is the number of contributors. After all, small, incremental changes may be a good thing in a language — though, spoiler alert: they are usually an indication of a series of larger issues, and we’ll come to that point later.

These are the number of developers over the same range of cycles, again split between committers to the full repository and to the full repository minus the vapi directory:

As you can see, the number of authors of changes is mostly stable, but still low. If we have few people that actively commit to the repository it means we have few people that can review a patch. It means patches linger longer and longer, while reviewers go through their queues; it means that contributors get discouraged; and, since nobody is paid to work full time on Vala, it means that any interruption caused by paid jobs will be a bottleneck on the project itself.

These concerns are not unique of a programming language: they exist for every volunteer-driven free and open source project. Programming languages, though, like core libraries, are problematic because any bottleneck causes ripple effects. You can take any stalled project you depend on, and vendor it into your own, but if that happens to the programming language you’re using, then you’re pretty much screwed.

For these reasons, we should also look at how well-distributed is the workload in Vala, i.e. which percentage of the work is done by the authors of those commits; the results are not encouraging. Over that range of cycles, Only two developers routinely crossed the 5% of commits:

Rico Tzschichholz
Jürg Billeter

And Rico has been the only one to consistently author >50% of the commits. This means there’s only one person dealing with the project on a day to day basis.

As the maintainer of a project who basically had to do all the work, I cannot even begin to tell you how soul-crushing that can become. You get burned out, and you feel responsible for everyone using your code, and then you get burned out some more. I honestly don’t want Rico to burn out, and you shouldn’t, either.

So, let’s go into unfair territory. These are the commits for Rust — the compiler and standard library:

These are the commits for Go — the compiler and base library:

These are the commits for Vala — both compiler and bindings:

These are the number of commits over the past year. Both languages are younger than Vala, have more tools than Vala, and are more used than Vala. Of course, it’s completely unfair to compare them, but those numbers should give you a sense of scale, of what is the current high bar for a successful programming language these days. Vala is a niche language, after all; it’s heavily piggy-backing on the GNOME community because it transpiles to C and needs a standard library and an ecosystem like the one GNOME provides. I never expected Vala to rise to the level of mindshare that Go and Rust currently occupy.

Nevertheless, we need to draw some conclusions about the current state of Vala — starting from this thread, perhaps, as it best encapsulates the issues the project is facing.

Vala, as a project, is limping along. There aren’t enough developers to actively effect change on the project; there aren’t enough developers to work on ancillary tooling — like build system integration, debugging and profiling tools, documentation. Saying that “Vala compiles to C so you can use tools meant for C” is comically missing the point, and it’s effectively like saying that “C compiles to binary code, so you can disassemble a program if you want to debug it”. Being able to inspect the language using tools native to the language is a powerful thing; if you have to do the name mangling in your head in order to set a breakpoint in GDB you are elevating the barrier of contributions way above the head of many newcomers.

Being able to effect change means also being able to introduce change effectively and without fear. This means things like continuous integration and a full test suite heavily geared towards regression testing. The test suite in Vala is made of 210 units, for a total of 5000 lines of code; the code base of Vala (vala AST, codegen, C code emitter, and the compiler) is nearly 75 thousand lines of code. There is no continuous integration, outside of the one that GNOME Continuous performs when building Vala, or the one GNOME developers perform when using jhbuild. Regressions are found after days or weeks, because developers of projects using Vala update their compiler and suddenly their projects cease to build.

I don’t want to minimise the enormous amount of work that every Vala contributor brought to the project; they are heroes, all of them, and they deserve as much credit and praise as we can give. The idea of a project-oriented, community-oriented programming language has been vindicated many times over, in the past 5 years.

If I scared you, or incensed you, then you can still blame me, and my lack of tact. You can still call me an asshole, and you can think that I’m completely uncool. What I do hope, though, is that this blog post pushes you into action. Either to contribute to Vala, or to re-new your commitment to it, so that we can look at my words in 5 years and say “boy, was Emmanuele wrong”; or to look at alternatives, and explore new venues in order to make GNOME (and the larger free software ecosystem) better.

Epoxy

2017-02-11T01:34:00+00:00

Epoxy is a small library that GTK+, and other projects, use in order to access the OpenGL API in somewhat sane fashion, hiding all the awful bits of craziness that actually need to happen because apparently somebody dosed the water supply at SGI with large quantities of LSD in the mid-‘90s, or something.

As an added advantage, Epoxy is also portable on different platforms, which is a plus for GTK+.

Since I’ve started using Meson for my personal (and some work-related) projects as well, I’ve been on the lookout for adding Meson build rules to other free and open source software projects, in order to improve both their build time and portability, and to improve Meson itself.

As a small, portable project, Epoxy sounded like a good candidate for the port of its build system from autotools to Meson.

To the Bat Build Machine!

tl;dr

Since you may be interested just in the numbers, building Epoxy with Meson on my Kaby Lake four Core i7 and NMVe SSD takes about 45% less time than building it with autotools.

A fairly good fraction of the autotools time is spent going through the autogen and configure phases, because they both aren’t parallelised, and create a ton of shell invocations.

Conversely, Meson’s configuration phase is incredibly fast; the whole Meson build of Epoxy fits in the same time the autogen.sh and configure scripts complete their run.

Administrivia

Epoxy is a simple library, which means it does not need a hugely complicated build system set up; it does have some interesting deviations, though, which made the porting an interesting challenge.

For instance, on Linux and similar operating systems Epoxy uses pkg-config to find things like the EGL availability and the X11 headers and libraries; on Windows, though, it relies on finding the opengl32 shared or static library object itself. This means that we get something straightforward in the former case, like:

# Optional dependencies
gl_dep = dependency('gl', required: false)
egl_dep = dependency('egl', required: false)

and something slightly less straightforward in the latter case:

if host_system == 'windows'
  # Required dependencies on Windows
  opengl32_dep = cc.find_library('opengl32', required: true)
  gdi32_dep = cc.find_library('gdi32', required: true)
endif

And, still, this is miles better than what you have to deal with when using autotools.

Let’s take a messy thing in autotools, like checking whether or not the compiler supports a set of arguments; usually, this involves some m4 macro that’s either part of autoconf-archive or some additional repository, like the xorg macros. Meson handles this in a much better way, out of the box:

# Use different flags depending on the compiler
if cc.get_id() == 'msvc'
  test_cflags = [
    '-W3',
    ...,
  ]
elif cc.get_id() == 'gcc'
  test_cflags = [
    '-Wpointer-arith',
    ...,
  ]
else
  test_cflags = [ ]
endif

common_cflags = []
foreach cflag: test_cflags
  if cc.has_argument(cflag)
    common_cflags += [ cflag ]
  endif
endforeach

In terms of speed, the configuration step could be made even faster by parallelising the compiler argument checks; right now, Meson has to do them all in a series, but nothing except some additional parsing effort would prevent Meson from running the whole set of checks in parallel, and gather the results at the end.

Generating code

In order to use the GL entry points without linking against libGL or libGLES* Epoxy takes the XML description of the API from the Khronos repository and generates the code that ends up being compiled by using a Python script to parse the XML and generating header and source files.

Additionally, and unlike most libraries in the G* stack, Epoxy stores its public headers inside a separate directory from its sources:

libepoxy
├── cross
├── doc
├── include
│   └── epoxy
├── registry
├── src
└── test

The autotools build has the src/gen_dispatch.py script create both the source and the header file for each XML at the same time using a rule processed when recursing inside the src directory, and proceeds to put the generated header under $(top_builddir)/include/epoxy, and the generated source under $(top_builddir)/src. Each code generation rule in the Makefile manually creates the include/epoxy directory under the build root to make up for parallel dispatch of each rule.

Meson makes is harder to do this kind of spooky-action-at-a-distance build, so we need to generate the headers in one pass, and the source in another. This is a bit of a let down, to be honest, and yet a build that invokes the generator script twice for each API description file is still faster under Ninja than a build with the single invocation under Make.

There are sill issues in this step that are being addressed by the Meson developers; for instance, right now we have to use a custom target for each generated header and source separately instead of declaring a generator and calling it multiple times. Hopefully, this will be fixed fairly soon.

Documentation

Epoxy has a very small footprint, in terms of API, but it still benefits from having some documentation on its use. I decided to generate the API reference using Doxygen, as it’s not a G* library and does not need the additional features of gtk-doc. Sadly, Doxygen’s default style is absolutely terrible; it would be great if somebody could fix it to make it look half as good as the look gtk-doc gets out of the box.

Cross-compilation and native builds

Now we get into “interesting” territory.

Epoxy is portable; it works on Linux and *BSD systems; on macOS; and on Windows. Epoxy also works on both Intel Architecture and on ARM.

Making it run on Unix-like systems is not at all complicated. When it comes to Windows, though, things get weird fast.

Meson uses cross files to determine the environment and toolchain of the host machine, i.e. the machine where the result of the build will eventually run. These are simple text files with key/value pairs that you can either keep in a separate repository, in case you want to share among projects; or you can keep them in your own project’s repository, especially if you want to easily set up continuous integration of cross-compilation builds.

Each toolchain has its own; for instance, this is the description of a cross compilation done on Fedora with MingW:

[binaries]
c = '/usr/bin/x86_64-w64-mingw32-gcc'
cpp = '/usr/bin/x86_64-w64-mingw32-cpp'
ar = '/usr/bin/x86_64-w64-mingw32-ar'
strip = '/usr/bin/x86_64-w64-mingw32-strip'
pkgconfig = '/usr/bin/x86_64-w64-mingw32-pkg-config'
exe_wrapper = 'wine'

This section tells Meson where the binaries of the MingW toolchain are; the exe_wrapper key is useful to run the tests under Wine, in this case.

The cross file also has an additional section for things like special compiler and linker flags:

[properties]
root = '/usr/x86_64-w64-mingw32/sys-root/mingw'
c_args = [ '-pipe', '-Wp,-D_FORTIFY_SOURCE=2', '-fexceptions', '--param=ssp-buffer-size=4', '-I/usr/x86_64-w64-mingw32/sys-root/mingw/include' ]
c_link_args = [ '-L/usr/x86_64-w64-mingw32/sys-root/mingw/lib' ]

These values are taken from the equivalent bits that Fedora provides in their MingW RPMs.

Luckily, the tool that generates the headers and source files is written in Python, so we don’t need an additional layer of complexity, with a tool built and run on a different platform and architecture in order to generate files to be built and run on a different platform.

Continuous Integration

Of course, any decent process of porting, these days, should deal with continuous integration. CI gives us confidence as to whether or not any change whatsoever we make actually works — and not just on our own computer, and our own environment.

Since Epoxy is hosted on GitHub, the quickest way to deal with continuous integration is to use TravisCI, for Linux and macOS; and Appveyor for Windows.

The requirements for Meson are just Python3 and Ninja; Epoxy also requires Python 2.7, for the dispatch generation script, and the shared libraries for GL and the native API needed to create a GL context (GLX, EGL, or WGL); it also optionally needs the X11 libraries and headers and Xvfb for running the test suite.

Since Travis offers an older version of Ubuntu LTS as its base system, we cannot build Epoxy with Meson; additionally, running the test suite is a crapshoot because the Mesa version if hopelessly out of date and will either cause most of the tests to be skipped or, worse, make them segfault. To sidestep this particular issue, I’ve prepared a Docker image with its own harness, and I use it as the containerised environment for Travis.

On Appveyor, thanks to the contribution of Thomas Marrinan we just need to download Python3, Python2, and Ninja, and build everything inside its own root; as an added bonus, Appveyor allows us to take the build artefacts when building from a tag, and shoving them into a zip file that gets deployed to the release page on GitHub.

Conclusion

Most of this work has been done off and on over a couple of months; the rough Meson build conversion was done last December, with the cross-compilation and native builds taking up the last bit of work.

Since Eric does not have any more spare time to devote to Epoxy, he was kind enough to give me access to the original repository, and I’ve tried to reduce the amount of open pull requests and issues there.

I’ve also released version 1.4.0 and I plan to do a 1.4.1 release soon-ish, now that I’m positive Epoxy works on Windows.

I’d like to thank:

Eric Anholt, for writing Epoxy and helping out when I needed a hand with it
Jussi Pakkanen and Nirbheek Chauhan, for writing Meson and for helping me out with my dumb questions on #mesonbuild
Thomas Marrinan, for working on the Appveyor integration and testing Epoxy builds on Windows
Yaron Cohen-Tal, for maintaining Epoxy in the interim

Constraints editing

2017-01-11T14:30:00+00:00

Last year I talked about the newly added support for Apple’s Visual Format Language in Emeus, which allows to quickly describe layouts using a cross between ASCII art and predicates. For instance, I can use:

H:|-[icon(==256)]-[name_label]-|
H:[surname_label]-|
H:[email_label]-|
H:|-[button(<=icon)]
V:|-[icon(==256)]
V:|-[name_label]-[surname_label]-[email_label]-|
V:[button]-|

and obtain a layout like this one:

Boxes approximate widgets

Thanks to the contribution of my colleague Martin Abente Lahaye, now Emeus supports extensions to the VFL, namely:

arithmetic operators for constant and multiplication factors inside predicates, like [button1(button2 * 2 + 16)]
explicit attribute references, like [button1(button1.height / 2)]

This allows more expressive layout descriptions, like keeping aspect ratios between UI elements, without requiring hitting the code base.

Of course, editing VFL descriptions blindly is not what I consider a fun activity, so I took some time to write a simple, primitive editing tool that lets you visualize a layout expressed through VFL constraints:

I warned you that it was primitive and simple

Here’s a couple of videos showing it in action:

At some point, this could lead to a new UI tool to lay out widgets inside Builder and/or Glade.

As of now, I consider Emeus in a stable enough state for other people to experiment with it — I’ll probably make a release soon-ish. The Emeus website is up to date, as it is the API reference, and I’m happy to review pull requests and feature requests.

Constraints (reprise)

2016-11-01T17:07:00+00:00

After the first article on Emeus various people expressed interest in the internals of the library, so I decided to talk a bit about what makes it work.

Generally, you can think about constraints as linear equations:

view1.attr1 = view2.attr2 × multiplier + constant

You take the the value of attr2 on the widget view2, multiply it by a multiplier, add a constant, and apply the value to the attribute attr1 on the widget view1. You don’t need view2.attr2 either, for instance:

view1.attr1 = constant

is a perfectly valid constraint.

You also don’t need to use an equality; these two constraints:

view1.width ≥ 180
view1.width ≤ 250

specify that the width of view1 must be in the [ 180, 250 ] range, extremes included.

Layout

A layout, then, is just a pile of linear equations that describe the relations between each element. So, if we have a simple grid:

+--------------------------------------------+
| super                                      |
|  +----------------+   +-----------------+  |
|  |     child1     |   |     child2      |  |
|  |                |   |                 |  |
|  +----------------+   +-----------------+  |
|                                            |
|  +--------------------------------------+  |
|  |               child3                 |  |
|  |                                      |  |
|  +--------------------------------------+  |
|                                            |
+--------------------------------------------+

We can describe each edge’s position and size using constraints. It’s important to note that there’s an implicit “reading” order that makes it easier to write constraints; in this case, we start from left to right, and from top to bottom. Generally speaking, it’s possible to describe constraints in any order, but the Cassowary solving algorithm is geared towards the “reading” order above.

Each layout has some implicit constraint already available. For instance, the “trailing” edge is equal to the leading edge plus the width; the bottom edge is equal to the top edge plus the height; the center point is equal to the width or height, divided by two, plus the leading or bottom edges. These constraints help solving the layout, as well as provide additional values to other constraints.

So, let’s start.

From the first row:

the leading edge of the super container is the same as the leading edge of child1, minus a padding
the trailing edge of child1 is the same as the leading edge of child2, minus a padding
the trailing edge of child2 is the same as the trailing edge of the super container, minus a padding
the width of child1 is the same as the width of child2

From the second row:

the leading edge of the super container is the same as the leading edge of child3, minus a padding
the trailing edge of child2 is the same as the trailing edge of the super container, minus a padding

From the first column:

the top edge of the super container is the same as the top edge of child1, minus a padding
the bottom edge of child1 is the same as the top edge of child3, minus a padding
the bottom edge of the super container is the same as the bottom edge of child3, minus a padding
the height of child3 is the same as the height of child1

From the second column:

the top edge of the super container is the same as the top edge of the child2, minus a padding
the bottom edge of child1 is the same as the top edge of child3, minus a padding
the bottom edge of the super container is the same as the bottom edge of child3, minus a padding
the height of child3 is the same as the height of child2

As you can see, there are some redundancies; these are necessary to ensure that the layout is fully resolved, though obviously there are some properties of the elements of the layout that implicitly eliminate some results. For instance, if child3‘s height is the same as child1, and child1 lies on the same row as child2 and it’s an axis-aligned rectangle, the it immediately follows that child3 must have the same height of child2 as well. It’s important to note that, from a solver perspective, there only are values, not boxes, and you could use the solver with any kind of geometric shape; only the constraints give us the information on what those shapes should be. It’s also easier to start from a fully constrained layout and then remove constraints, than to start from a loosely constrained layout and add constraints until it’s stable.

Representation

From the text description we can now get into a system of equations:

super.start = child1.start - padding
child1.end = child2.start - padding
super.end = child2.end - padding
child1.width = child2.width
super.start = child3.start - padding
super.end = child3.end - padding
super.top = child1.top - padding
child1.bottom = child3.top - padding
super.bottom = child3.bottom - padding
child3.height = child1.height
super.top = child2.top - padding
child2.bottom = child3.top - padding
child3.height = child2.height

Apple, in its infinite wisdom and foresight, decided that this form is still too verbose. After looking at the Perl format page for far too long, Apple engineers came up with the Visual Format Language, or VFL for short.

Using VFL, the constraints above become:

H:|-(padding)-[child1(==child2)]-(padding)-[child2]-(padding)-|
H:|-(padding)-[child3]-(padding)-|
V:|-(padding)-[child1(==child3)]-(padding)-[child3]-(padding)-|
V:|-(padding)-[child2(==child3)]-(padding)-[child3]-(padding)-|

Emeus, incidentally, ships with a simple utility that can take a set of VFL format strings and generate GtkBuilder descriptions that you can embed into your templates.

Change

We’ve used a fair amount of constraints, or four lines of faily cryptic ASCII art, to basically describe a non-generic GtkGrid with two equally sized horizontal cells on the first row, and a single cell with a column span of two; compared to the common layout managers inside GTK+, this does not seem like a great trade off.

Except that we can describe any other layout without necessarily having to pack widgets inside boxes, with margins and spacing and alignment rules; we also don’t have to change the hierarchy of the boxes if we want to change the layout. For instance, let’s say that we want child3 to have a different horizontal padding, and a minimum and maximum width; we just need to change the constraints involved in that row:

H:|-(hpadding)-[child3(>=250,<=500)]-(hpadding)-|

Additionally, we now want to decouple child1 and child3 heights, and make child1 a fixed height item:

V:|-(padding)-[child1(==250)]-(padding)-[child3]-(padding)-|

And make the height of child3 move within a range of values:

V:|-(padding)-[child2]-(padding)-[child3(>=200,<=300)]-(padding)-|

For all these cases we’d have to add intermediate boxes in between our children and the parent container — with all the issues of theming and updating things like GtkBuilder XML descriptions that come with that.

Future

The truth is, though, that describing layouts in terms of constraints is another case of software engineering your way out of talking with designers; it’s great to start talking about incremental simplex solvers, and systems of linear equations, and ASCII art to describe your layouts, but it doesn’t make UI designers really happy. They can deal with it, and having a declarative language to describe constraints is more helpful than parachuting them into an IDE with a Swiss army knife and a can of beans, but I wouldn’t recommend it as a solid approach to developer experience.

Havoc wrote a great article on how layout management API doesn’t necessarily have to suck:

we can come up with a better, descriptive API that does not make engineers and designers cringe in different ways
we should have support from our tools, in order to manipulate constraints and UI elements
we should be able to combine boxes (which are easy to style) and constraints (which are easy to lay out) together in a natural and flexible way

Improving layout management should be a goal in the development of GTK+ 4.0, so feel free to jump in and help out.

Constraints

2016-10-17T18:30:00+01:00

GUI toolkits have different ways to lay out the elements that compose an application’s UI. You can go from the fixed layout management — somewhat best represented by the old ‘90s Visual tools from Microsoft; to the “springs and struts” model employed by the Apple toolkits until recently; to the “boxes inside boxes inside boxes” model that GTK+ uses to this day. All of these layout policies have their own distinct pros and cons, and it’s not unreasonable to find that many toolkits provide support for more than one policy, in order to cater to more use cases.

For instance, while GTK+ user interfaces are mostly built using nested boxes to control margins, spacing, and alignment of widgets, there’s a sizeable portion of GTK+ developers that end up using GtkFixed or GtkLayout containers because they need fixed positioning of children widget — until they regret it, because now they have to handle things like reflowing, flipping contents in right-to-left locales, or font size changes.

Additionally, most UI designers do not tend to “think with boxes”, unless it’s for Web pages, and even in that case CSS affords a certain freedom that cannot be replicated in a GUI toolkit. This usually results in engineers translating a UI specification made of ties and relations between UI elements into something that can be expressed with a pile of grids, boxes, bins, and stacks — with all the back and forth, validation, and resources that the translation entails.

It would certainly be easier if we could express a GUI layout in the same set of relationships that can be traced on a piece of paper, a UI design tool, or a design document:

this label is at 8px from the leading edge of the box
this entry is on the same horizontal line as the label, its leading edge at 12px from the trailing edge of the label
the entry has a minimum size of 250px, but can grow to fill the available space
there’s a 90px button that sits between the trailing edge of the entry and the trailing edge of the box, with 8px between either edges and itself

Sure, all of these constraints can be replaced by a couple of boxes; some packing properties; margins; and minimum preferred sizes. If the design changes, though, like it often does, reconstructing the UI can become arbitrarily hard. This, in turn, leads to pushback to design changes from engineers — and the cost of iterating over a GUI is compounded by technical inertia.

For my daily work at Endless I’ve been interacting with our design team for a while, and trying to get from design specs to applications more quickly, and with less inertia. Having CSS available allowed designers to be more involved in the iterative development process, but the CSS subset that GTK+ implements is not allowed — for eminently good reasons — to change the UI layout. We could go “full Web”, but that comes with a very large set of drawbacks — performance on low end desktop devices, distribution, interaction with system services being just the most glaring ones. A native toolkit is still the preferred target for our platform, so I started looking at ways to improve the lives of UI designers with the tools at our disposal.

Expressing layout through easier to understand relationships between its parts is not a new problem, and as such it does not have new solutions; other platforms, like the Apple operating systems, or Google’s Android, have started to provide this kind of functionality — mostly available through their own IDE and UI building tools, but also available programmatically. It’s even available for platforms like the Web.

What many of these solutions seem to have in common is using more or less the same solving algorithm — Cassowary.

Cassowary is:

an incremental constraint solving toolkit that efficiently solves systems of linear equalities and inequalities. Constraints may be either requirements or preferences. Client code specifies the constraints to be maintained, and the solver updates the constrained variables to have values that satisfy the constraints.

This makes it particularly suited for user interfaces.

The original implementation of Cassowary was written in 1998, in Java, C++, and Smalltalk; since then, various other re-implementations surfaced: Python, JavaScript, Haskell, slightly-more-modern-C++, etc.

To that collection, I’ve now added my own — written in C/GObject — called Emeus, which provides a GTK+ container and layout manager that uses the Cassowary constraint solving algorithm to compute the allocation of each child.

In spirit, the implementation is pretty simple: you create a new EmeusConstraintLayout widget instance, add a bunch of widgets to it, and then use EmeusConstraint objects to determine the relations between children of the layout:

simple-grid.js[Lines 89-170]download

        let button1 = new Gtk.Button({ label: 'Child 1' });
        this._layout.pack(button1, 'child1');
        button1.show();

        let button2 = new Gtk.Button({ label: 'Child 2' });
        this._layout.pack(button2, 'child2');
        button2.show();

        let button3 = new Gtk.Button({ label: 'Child 3' });
        this._layout.pack(button3, 'child3');
        button3.show();

        this._layout.add_constraints([
            new Emeus.Constraint({ target_attribute: Emeus.ConstraintAttribute.START,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button1,
                                   source_attribute: Emeus.ConstraintAttribute.START,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_object: button1,
                                   target_attribute: Emeus.ConstraintAttribute.WIDTH,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button2,
                                   source_attribute: Emeus.ConstraintAttribute.WIDTH }),
            new Emeus.Constraint({ target_object: button1,
                                   target_attribute: Emeus.ConstraintAttribute.END,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button2,
                                   source_attribute: Emeus.ConstraintAttribute.START,
                                   constant: -12.0 }),
            new Emeus.Constraint({ target_object: button2,
                                   target_attribute: Emeus.ConstraintAttribute.END,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_attribute: Emeus.ConstraintAttribute.END,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_attribute: Emeus.ConstraintAttribute.START,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button3,
                                   source_attribute: Emeus.ConstraintAttribute.START,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_object: button3,
                                   target_attribute: Emeus.ConstraintAttribute.END,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_attribute: Emeus.ConstraintAttribute.END,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_attribute: Emeus.ConstraintAttribute.TOP,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button1,
                                   source_attribute: Emeus.ConstraintAttribute.TOP,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_attribute: Emeus.ConstraintAttribute.TOP,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button2,
                                   source_attribute: Emeus.ConstraintAttribute.TOP,
                                   constant: -8.0 }),
            new Emeus.Constraint({ target_object: button1,
                                   target_attribute: Emeus.ConstraintAttribute.BOTTOM,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button3,
                                   source_attribute: Emeus.ConstraintAttribute.TOP,
                                   constant: -12.0 }),
            new Emeus.Constraint({ target_object: button2,
                                   target_attribute: Emeus.ConstraintAttribute.BOTTOM,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button3,
                                   source_attribute: Emeus.ConstraintAttribute.TOP,
                                   constant: -12.0 }),
            new Emeus.Constraint({ target_object: button3,
                                   target_attribute: Emeus.ConstraintAttribute.HEIGHT,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button1,
                                   source_attribute: Emeus.ConstraintAttribute.HEIGHT }),
            new Emeus.Constraint({ target_object: button3,
                                   target_attribute: Emeus.ConstraintAttribute.HEIGHT,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_object: button2,
                                   source_attribute: Emeus.ConstraintAttribute.HEIGHT }),
            new Emeus.Constraint({ target_object: button3,
                                   target_attribute: Emeus.ConstraintAttribute.BOTTOM,
                                   relation: Emeus.ConstraintRelation.EQ,
                                   source_attribute: Emeus.ConstraintAttribute.BOTTOM,
                                   constant: -8.0 }),
        ]);

A simple grid

This obviously looks like a ton of code, which is why I added the ability to describe constraints inside GtkBuilder XML:

centered.ui[Lines 28-45]download

            <constraints>
              <constraint target-object="button_child"
                          target-attr="center-x"
                          relation="eq"
                          source-object="super"
                          source-attr="center-x"
                          strength="required"/>
              <constraint target-object="button_child"
                          target-attr="EMEUS_CONSTRAINT_ATTRIBUTE_CENTER_Y"
                          relation="eq"
                          source-object="super"
                          source-attr="center-y"/>
              <constraint target-object="button_child"
                          target-attr="width"
                          relation="ge"
                          constant="200"
                          strength="EMEUS_CONSTRAINT_STRENGTH_STRONG"/>
            </constraints>

Additionally, I’m writing a small parser for the Visual Format Language used by Apple for their own auto layout implementation — even though it does look like ASCII art of Perl format strings, it’s easy to grasp.

The overall idea is to prototype UIs on top of this, and then take advantage of GTK+’s new development cycle to introduce something like this and see if we can get people to migrate from GtkFixed/GtkLayout.

Who wrote GTK+ and more

2016-09-21T11:19:00+01:00

I’ve just posted on the GTK+ development blog the latest article in the “who writes GTK+” series. Now that we have a proper development blog, this is the kind of content that should be present there instead of my personal blog.

If you’re not following the GTK+ development blog, you really should do so now. Additionally, you should follow @GTKToolkit on Twitter and, if you’re into niche social platforms, a kernel developer, or a Linus groupie, there’s the GTK+ page on Google Plus.

Additionally, if you’re contributing to GTK+, or using it, you should consider writing an article for the developers blog; just contact me on IRC (I’m ebassi on irc.gnome.org) or send me an email if you have an idea about something you worked on, a new feature, some new API, or the internals of the platform.

GSK Demystified (III) — Interlude

2016-08-27T10:28:00+01:00

See the the tag for the GSK demystified series for the other articles in the series.

There have been multiple reports after GUADEC about the state of GSK, so let’s recap a bit by upholding the-long standing tradition of using a FAQ format as a rhetorical device.

Q: Is GSK going to be merged in time for 3.22?

A: Short answer: no.

Long-ish answer: landing a rewrite of how GTK renders its widgets near the end of a stable API cycle, when a bunch of applications that tend to eschew GTK+ itself for rendering their content — like Firefox or LibreOffice — finally, after many years, ported to GTK+ 3, seemed a bit sadistic on our part.

Additionally, GSK still has some performance issues when it comes to large or constantly updating UIs; try running, for instance gtk3-widget-factory on HiDPI using the wip/ebassi/gsk-renderer branch and marvel at the 10 fps we achieve currently.

Q: Aside from performance, are there any other concerns?

A: Performance is pretty much the biggest concern we found. We need to reduce the amount of rasterizations we perform with Cairo, and we need better ways to cache and reuse those rasterizations across frames; we really want all buttons with the same CSS state and size to be rasterized once, for instance, and just drawn multiple times in their right place. The same applies to things like icons. Caching text runs and glyphs would also be a nice win.

The nice bit is that, with a fully retained render tree, now we can actually do this.

The API seems to have survived contact with the widget drawing code inside GTK+, so it’s a matter of deciding how much we need to provide in terms of convenience API for out-of-tree widgets and containers. The fallback code is in place, right now, which means that porting widgets can proceed at its own pace.

There are a few bugs in the rendering code, like blend modes; and I still want to have filters like blur and color conversions in the GskRenderNode API.

Finally, there’s still the open question of the mid-level scene graph API, or GskLayer, that will replace Clutter and Clutter-GTK; the prototype is roughly done, but things like animations are not set in stone due to lack of users.

Q: Is there a plan for merging GSK?

A: Yes, we do have a plan.

The merge window mostly hinges on when we’re going to start with a new development cycle for the next API, but we decided that as soon as the window opens, GSK will land. Ideally we want to ensure that, by the time 4.0 rolls around, there won’t be any users of GtkWidget::draw left inside GNOME, so we’ll be able to deprecate its use, and applications targeting the new stable API will be able to port away from it.

Having a faster, more featureful, and more optimized rendering pipeline inside GTK+ is a pretty good new feature for the next API cycle, and we think that the port is not going to be problematic, given the amount of fallback code paths in place.

Additionaly, by the time we release GTK+ 4.0, we’ll have a more battle-tested API to replace Clutter and Clutter-GTK, allowing applications to drop a dependency.

Q: How can I help?

A: If you’re a maintainer of a GTK+ library or application, or if you want to help out the development of GTK+ itself, then you can pick up my GSK development branch, fork it off, and look at porting widgets and containers. I’m particularly interested in widgets using complex drawing operations. See where the API is too bothersome, and look for patterns we can wrap into convenience API provided by GTK+ itself. For instance, the various gtk_render_* family of functions are a prime candidate for being replaced by equivalent functions that return a GskRenderNode instead.

Testing is also welcome; for instance, look at missing widgets or fragments of rendering.

Hopefully, the answers above should have provided enough context for the current state of GSK.

The next time, we’ll return to design and implementation notes of the API itself.

GUADEC/2

2016-08-18T09:00:00+01:00

Writing this from my home office, while drinking the first coffee of the day.

Once again, GUADEC has come and gone.

Once again, it was impeccably organized by so many wonderful volunteers.

Once again, I feel my batteries recharged.

Once again, I’ve had so many productive conversations.

Once again, I’ve had many chances to laugh.

Once again, I’ve met both new and long since friends.

Once again, I’ve visited a new city, with interesting life, food, drinks, and locations.

Once again, thanks to everybody who worked very hard to make this GUADEC happen and be the success it was.

Once again, we return.

GUADEC

2016-08-10T15:40:00+01:00

I’m going to talk about the evolution of GTK+ rendering, from its humble origins of X11 graphics contexts, to Cairo, to GSK. If you are interested in this kind of stuff, you can either attend my presentation on Saturday at 11 in the Grace Room, or you can just find me and have a chat.

I’m also going to stick around during the BoF days — especially for the usual GTK+ team meeting, which will be on the 15th.

See you all in Karlsruhe.

GSK Demystified (II) — Rendering

2016-08-10T15:20:00+01:00

See the previous article for an introduction to GSK.

In order to render with GSK we need to get acquainted with two classes:

GskRenderNode, a single element in the rendering tree
GskRenderer, the object that effectively turns the rendering tree into rendering commands

GskRenderNode

The usual way to put things on the screen involves asking the windowing system to give us a memory region, filling it with something, and then asking the windowing system to present it to the graphics hardware, in the hope that everything ends up on the display. This is pretty much how every windowing system works. The only difference lies in that “filling it with something”.

With Cairo you get a surface that represents that memory region, and a (stateful) drawing context; every time you need to draw you set up your state and emit a series of commands. This happens on every frame, starting from the top level window down into every leaf object. At the end of the frame, the content of the window is swapped with the content of the buffer. Every frame is drawn while we’re traversing the widget tree, and we have no control on the rendering outside of the state of the drawing context.

A tree of GTK widgets

With GSK we change this process with a small layer of indirection; every widget, from the top level to the leaves, creates a series of render nodes, small objects that each hold the drawing state for their contents. Each node is, at its simplest, a collection of:

a rectangle, representing the region used to draw the contents
a transformation matrix, representing the parent-relative set of transformations applied to the contents when drawing
the contents of the node

Every frame, thus, is composed of a tree of render nodes.

A tree of GTK widgets and GSK render nodes

The important thing is that the render tree does not draw anything; it describes what to draw (which can be a rasterization generated using Cairo) and how and where to draw it. The actual drawing is deferred to the GskRenderer instance, and will happen only once the tree has been built.

After the rendering is complete we can discard the render tree. Since the rendering is decoupled from the widget state, the widgets will hold all the state across frames — as they already do. Each GskRenderNode instance is, thus, a very simple instance type instead of a full GObject, whose lifetime is determined by the renderer.

GskRenderer

The renderer is the object that turns a render tree into the actual draw commands. At its most basic, it’s a simple compositor, taking the content of each node and its state and blending it on a rendering surface, which then gets pushed to the windowing system. In practice, it’s a tad more complicated than that.

Each top-level has its own renderer instance, as it requires access to windowing system resources, like a GL context. When the frame is started, the renderer will take a render tree and a drawing context, and will proceed to traverse the render tree in order to translate it into actual render commands.

As we want to offload the rendering and blending to the GPU, the GskRenderer instance you’ll most likely get is one that uses OpenGL to perform the rendering. The GL renderer will take the render tree and convert it into a (mostly flat) list of data structures that represent the state to be pushed on the state machine — the blending mode, the shading program, the textures to sample, and the vertex buffer objects and attributes that describe the rendering. This “translation” stage allows the renderer to decide which render nodes should be used and which should be discarded; it also allows us to create, or recycle, all the needed resources when the frame starts, and minimize the state transitions when doing the actual rendering.

Going from here to there

Widgets provided by GTK will automatically start using render nodes instead of rendering directly to a Cairo context.

There are various fallback code paths in place in the existing code, which means that, luckily, we don’t have to break any existing out of tree widget: they will simply draw themselves (and their children) on an implicit render node. If you want to port your custom widgets or containers, on the other hand, you’ll have to remove the GtkWidget::draw virtual function implementation or signal handler you use, and override the GtkWidget::get_render_node() virtual function instead.

Containers simply need to create a render node for their own background, border, or custom drawing; then they will have to retrieve the render node for each of their children. We’ll provide convenience API for that, so the chances of getting something wrong will be, hopefully, reduced to zero.

Leaf widgets can remain unported a bit longer, unless they are composed of multiple rendering elements, in which case they simply need to create a new render node for each element.

I’ll provide more example of porting widgets in a later article, as soon as the API will have stabilized.

GSK Demystified (I) — A GSK primer

2016-07-05T11:11:00+01:00

Last month I published an article on how GTK+ draws widgets on the toolkit development blog. The article should give you some background on the current state of what GTK does when something asks it to draw what you see on the screen — so it’s probably a good idea to read that first, and then come back here. Don’t worry, I’ll wait…

Welcome back! Now that we’re on the same page… What I didn’t say in that article is that most of it happens on your CPU, rather than on your GPU — except the very last step, when the compositor takes the contents of each window and pushes them to the GPU, likely via the 3D pipeline provided by your windowing system, to composite them into what you’ll likely see on your screen.

The goal for GUI toolkits, for the past few years, has been to take advantage of the GPU programmable pipeline as much as possible, as it allows to use the right hardware for the job, while keeping your CPU free for working on the application logic, or simply powered down and avoid polar bears to squeeze on an ever reducing sheet of artic ice. It also allows to improve the separation of jobs internally to the toolkit, with the potential of splitting up the work across multiple CPU cores.

As toolkit developers, we currently have only one major API for talking to the GPU, programming it, and using it to put the contents of a window on the screen, and that’s OpenGL.

You may think: well, we use Cairo; Cairo has support for an OpenGL device. Just enable that, and we’re good to go, right? and you wouldn’t be entirely wrong — except that you really don’t want to use the OpenGL Cairo device in production, as it’s both a poor fit for the Cairo drawing model and it’s basically unmaintained. Also, Cairo is pretty much 2D only, and while you can fake some 3D transformations with it, it’s definitely not up to the task of implementing the full CSS transformation specification.

Using OpenGL to generate pixel-perfect results is complicated, and in some cases it just goes against the expectations of the GPU itself: reading back data; minuscule fragments and tesselations; tons of state changes — those are all pretty much no-go areas when dealing with a GPU.

On the other hand, we really want to stop relying so much on the CPU for drawing; leaving your cores idle allows them to go into low power states, preserving them and improving your battery life; additionally, any cycle that is not spent inside the toolkit is a cycle available to your application logic.

As you may know from the past few years, I’ve been working on writing a new API that lets GTK offload to the GPU what currently happens on the CPU; it’s called GSK — short for GTK Scene Kit — and its meant to achieve two things:

render the contents of a GTK application more efficiently
provide a scene graph API to both the toolkit and applications

With these two goals in mind, I want to give a quick overview on how GSK works, and at which point we are in the development.

As GSK is meant to serve two purposes it makes sense to have two separate layers of API. This is a design decision that solidified after various discussions at GUADEC 2015. As such, it required a fair amount of rework of the existing code base, but very much for the better.

At the lowest level we have:

GskRenderNode, which is used to describe a tree of textures, blend modes, filters, and transformations; this tree is easily converted in render operations for graphics API like Cairo and OpenGL, and Vulkan in the near future.
GskRenderer, an object that takes a tree of GskRenderNode instances that describes the contents of a frame, and renders it on a given GdkDrawingContext.

Every time you wish to render something, you build a tree of render nodes; specify their content; set up their transformations, opacity, and blending; and, finally, you pass the tree to the renderer. After that, the renderer owns the render nodes tree, so you can safely discard it after each frame.

On top of this lower level API we can implement both the higher level scene graph API based on GskLayer that I presented at GUADEC; and GTK+ itself, which allows us to avoid reimplementing GTK+ widgets in terms of GSK layers.

I’m going to talk about GskRenderer and GskRenderNode more in depth in a future blog post, but if you’re looking for some form of prior art, you can check the ClutterPaintNode API in Clutter.

Widgets in GTK+ would not really be required to use render nodes: ideally, we want to get to a future where widgets are a small, composable unit whose appearances that can be described using CSS; while we build towards that future, though, we can incrementally transition from the current immediate more rendering model to a more structured tree of rendering operations that can be reordered and optimized for the target graphics layer.

Additionally, by sharing the same rendering model between the more complex widget API and the more freeform layers one, we only have to care about optmizing a single set of operations.

You can check the current progress of my work in the gsk-renderer branch of the GTK+ repository.

Long term support for GTK+

2016-06-15T16:20:00+01:00

Dear Morten,

A belief that achieving stability can be done after most of the paid contributors have run off to play with new toys is delusional. The record does not support it.

The record (in terms of commit history) seems to not support your position — as much as you think everyone else is “delusional” about it, the commit log does not really lie.

The 2.24.0 release was cut in January, 2011 — five and half years ago. No new features, no new API. Precisely what would happen with the new release plan, except that the new plan would also give a much better cadence to this behaviour.

Since then, the 2.24 branch — i.e. the “feature frozen” branch has seen 873 commits (as of this afternoon, London time), and 30 additional releases.

Turns out that people are being paid to maintain feature-frozen branches because that’s where the “boring” bits are — security issues, stability bugs, etc. Volunteers are much more interested in getting the latest and greatest feature that probably does not interest you now, but may be requested by your users in two years.

Isn’t it what you asked multiple times? A “long term support” release that gives you time to port your application to a stable API that has seen most of the bugs and uncertainty already squashed?

Experiments in Meson

2016-06-09T00:00:00+01:00

Last GUADEC I attended Jussi Pakkanen’s talk about his build system, Meson; if you weren’t there, I strongly recommend you watch the recording. I left the talk impressed, and I wanted to give Meson a try. Cue 9 months later, and a really nice blog post from Nirbheek on how Centricular is porting GStreamer from autotools to Meson, and I decided to spend some evening/weekend time on learning Meson.

I decided to use the simplest project I maintain, the one with the minimal amount of dependencies and with a fairly clean autotools set up — i.e. Graphene.

Graphene has very little overhead in terms of build system by itself; all it needs are:

a way to check for compiler flags
a way to check for the existence of headers and types
a way to check for platform-specific extensions, like SSE or NEON

Additionally, it needs a way to generate documentation and introspection data, but those are mostly hidden in weird incantations provided by other projects, like gtk-doc and gobject-introspection, so most of the complexity is hidden from the maintainer (and user) point of view.

Armed with little more than the Meson documentation wiki and the GStreamer port as an example, I set off towards the shining new future of a small, sane, fast build system.

The Good

Meson uses additional files, so I didn’t have to drop the autotools set up while working on the Meson one. Once I’m sure that the results are the same, I’ll be able to remove the various configure.ac, Makefile.am, and friends, and leave just the Meson file.

Graphene generates two header files during its configuration process:

a config.h header file, for internal use; we use this file to check if a specific feature or header is available while building Graphene itself
a graphene-config.h header file, for public use; we expose this file to Graphene users for build time detection of platform features

While the autotools code that generates config.h is pretty much hidden from the developer perspective, with autoconf creating a template file for you by pre-parsing the build files, the part of the build system that generates the graphene-config.h one is pretty much a mess of shell script, cacheable variables for cross-compilation, and random m4 escaping rules. Meson, on the other hand, treats both files exactly the same way: generate a configuration object, set variables on it, then take the appropriate configuration object and generate the header file — with or without a template file as an input:

# Internal configuration header
configure_file(input: 'config.h.meson',
               output: 'config.h',
               configuration: conf)

# External configuration header
configure_file(input: 'graphene-config.h.meson',
               output: 'graphene-config.h',
               configuration: graphene_conf,
               install: true,
               install_dir: 'lib/graphene-1.0/include')

While explicit is better than implicit, at least most of the time, having things taken care for you avoids the boring bits and, more importantly, avoids getting the boring bits wrong. If I had a quid for every broken invocation of the introspection scanner I’ve ever seen or had to fix, I’d probably retire on a very small island. In Meson, this is taken care by a function in the gnome module:

    import('gnome')

    # Build introspection only if we enabled building GObject types
    build_gir = build_gobject
    if build_gobject and get_option('enable-introspection')
      gir = find_program('g-ir-scanner', required: false)
      build_gir = gir.found() and not meson.is_cross_build()
    endif

    if build_gir
      gir_extra_args = [
        '--identifier-filter-cmd=' + meson.source_root() + '/src/identfilter.py',
        '--c-include=graphene-gobject.h',
        '--accept-unprefixed',
        '-DGRAPHENE_COMPILATION',
        '--cflags-begin',
        '-I' + meson.source_root() + '/src',
        '-I' + meson.build_root() + '/src',
        '--cflags-end'
      ]
      gnome.generate_gir(libgraphene,
                         sources: headers + sources,
                         namespace: 'Graphene',
                         nsversion: graphene_api_version,
                         identifier_prefix: 'Graphene',
                         symbol_prefix: 'graphene',
                         export_packages: 'graphene-gobject-1.0,
                         includes: [ 'GObject-2.0' ],
                         install: true,
                         extra_args: gir_extra_args)
    endif

Meson generates Ninja rules by default, and it’s really fast at that. I can get a fully configured Graphene build set up in less that a couple of seconds. On top of that, Ninja is incredibly fast. The whole build of Graphene takes less than 5 seconds — and I’m counting building the tests and benchmarks, something that I had to move to be on demand for the autotools set up because they added a noticeable delay to the build. Now I always know if I’ve just screwed up the build, and not just when I run make check.

Jussi is a very good maintainer, helpful and attentive at issues reported to his project, and quick at reviewing patches. The terms for contributing to Meson are fairly standard, and the barrier for entry is very low. For a project like a build system, which interacts and enables other projects, this is a very important thing.

The Ugly

As I said, Meson has some interesting automagic handling of the boring bits of building software, like the introspection data. But there are other boring bits that do not have convenience wrappers, and thus you get into overly verbose section of your meson.build — and while it’s definitely harder to get those wrong, compared to autoconf or automake, it can still happen.

Even in the case of automagic handling, though, there are cases when you have to deal with some of the magic escaping from under the rug. Generally it’s not hard to understand what’s missing or what’s necessary, but it can be a bit daunting when you’re just staring at a Python exception barfed on your terminal.

The documentation is kept in a wiki, which is generally fine for keeping it up to date; but it’s hard to search — as all wikis are — and hard to visually scan. I’ve lost count of the times I had to search for all the methods on the meson built-in object, and I never remember which page I have to search for, or in.

The inheritance chain for some objects is mentioned in passing, but it’s hard to track; which methods does the test object have? What kind of arguments does the compiler.compiles() method have? Are they positional or named?

The syntax and API reference documentation should probably be generated from the code base, and look more like an API reference than a wiki.

Examples are hard to come by. I looked at the GStreamer port, but I also had to start looking at Meson’s own test suite.

Modules are all in tree, at least for the time being. This means that if I want to add an ad hoc module for a whole complex project like, say, GNOME, I’d have to submit it to upstream. Yeah, I know: bad example, Meson already has a GNOME module; but the concept still applies.

Meson does not do dist tarballs. I’ve already heard people being skeptical about this point, but I personally don’t care that much. I can generate a tarball from a Git tag, and while it won’t be self-hosting, it’s already enough to get a distro going. Seriously, though: building from a Git tag is a better option than building from a tarball, in 2016.

The Bad

There shocking twist is that nothing stands out as “bad”. Mostly, it’s just ugly stuff — caused either by missing convenience functionality that will by necessity appear once people start using Meson more; or by the mere fact that all build systems are inherently ugly.

On the other hand, there’s some badness in the tooling around project building. For instance, Travis-CI does not support it, mostly because they use an ancient version of Ubuntu LTS as the base environment. Jhbuild does not have a Meson/Ninja build module, so we’ll have to write that one; same thing for GNOME Builder. While we wait, having a dummy configure script or a dummy Makefile that would probably help.

These are not bad things per se, but they definitely block further adoption.

tl;dr

I think Meson has great potential, and I’d love to start using it more for my projects. If you’re looking for a better, faster, and more understandable build system then you should grab Meson and explore it.

Reviving the GTK development blog

2016-05-16T00:00:00+01:00

The GTK+ project has a development blog.

I know it may come as a shock to many of you, and you’d be completely justified in thinking that I just made that link up — but the truth is, the GTK+ project has had a development blog for a long while.

Sadly, the blog hasn’t been updated in five years — mostly around the time 3.0 was released, and the GTK+ website was revamped; even before that, the blog was mostly used for release announcements, which do not make for very interesting content.

Like many free and open source software projects, GTK+ has various venues of interaction between its contributors and its users; mailing lists, personal blogs, IRC, Stack Overflow, reddit, and many, many other channels. In this continuum of discussions it’s both easy to get lost and to lose the sense of having said things before — after all, if I repeat something at least three times a week on three different websites for three years, how can people still not know about it? Some users will always look at catching up after three years, because their projects live on very different schedules that the GTK releases one; others will try to look for official channels, even if the free and open source software landscape has fragmented to such a degree that any venue can be made “official” by the simple fact of having a contributor on it; others again will look at the API reference for any source of truth, forgetting, possibly, that if everything went into the API reference then it would cease to be useful as a reference.

The GTK+ development blog is not meant to be the only source for truth, or the only “official” channel; it’s meant to be a place for interesting content regarding the project, for developers using GTK+ or considering to use it; a place that acts as a hub to let interested people discover what’s up with GTK+ itself but that don’t want to subscribe to the commits list or join IRC.

From an editorial standpoint, I’d like the GTK+ development blog to be open to contribution from people contributing to GTK+; using GTK+; and newcomers to the GTK+ code base and their experiences. What’s a cool GTK+ feature that you worked on? How did GTK+ help you in writing your application or environment? How did you find contributing to GTK+ for the first time? If you want to write an article for the GTK+ blog talking about this, then feel free to reach out to me with an outline, and I’ll be happy to help you.

In the meantime, the first post in the This Week in GTK+ series has gone up; you’ll get a new post about it every Monday, and if you want to raise awareness on something that happened during a week, feel free to point it out on the wiki.

Who wrote GTK+ 3.20

2016-05-06T00:00:00+01:00

Last time I tried to dispel the notion that GTK+ is dead or dying. Others have also chimed in, and it seems that we’re picking up the pace into making GTK a more modern, more useful driving force into the Linux desktop ecosystem.

Let’s see how much has changed in the six months of the 3.20 development cycle.

Once again, to gather the data, I’ve used the most excellent git-dm tool that Jonathan Corbet wrote for the “Who wrote the Linux kernel” columns for LWN. As usual, I’ve purposefully skipped the commits dealing with translations, to avoid messing up the statistics.

You should look at my previous article as a comparison point.

Activity

For the 3.20 cycle, the numbers are:

	Version	Lines added	Lines removed	Delta	Contributors
GLib	2.48	20597	7544	13053	55
GTK+	3.20	158427	117823	40604	81

More or less stable in terms of contributors, but as you can see the number of lines added and removed has doubled. This is definitely the result of the changes in the CSS machinery that have (finally) brought it to a stable as well as more featureful state.

Contributors

GLib

Of the 55 developers that contributed the 271 changesets of GLib during the 3.20 development cycle, the most active are:

Name	Per changeset	Name	Per changed lines
Ignacio Casal Quinteiro	56 (20.7%)	Ignacio Casal Quinteiro	8530 (39.7%)
Philip Withnall	42 (15.5%)	Philip Withnall	5402 (25.1%)
Allison Ryan Lortie	27 (10.0%)	Matthias Clasen	3228 (15.0%)
Chun-wei Fan	22 (8.1%)	Chun-wei Fan	1440 (6.7%)
Matthias Clasen	18 (6.6%)	Allison Ryan Lortie	1338 (6.2%)
Dan Winship	9 (3.3%)	Javier Jardón	565 (2.6%)
Mikhail Zabaluev	7 (2.6%)	Iain Lane	149 (0.7%)
Marc-André Lureau	6 (2.2%)	Ruslan Izhbulatov	147 (0.7%)
Ruslan Izhbulatov	6 (2.2%)	Dan Winship	95 (0.4%)
Rico Tzschichholz	6 (2.2%)	Lars Uebernickel	79 (0.4%)
Xavier Claessens	6 (2.2%)	Xavier Claessens	74 (0.3%)
Emmanuele Bassi	5 (1.8%)	Christian Hergert	71 (0.3%)
Iain Lane	4 (1.5%)	Mikhail Zabaluev	48 (0.2%)
Lars Uebernickel	3 (1.1%)	Rico Tzschichholz	45 (0.2%)
Sébastien Wilmet	3 (1.1%)	Daiki Ueno	42 (0.2%)
Simon McVittie	3 (1.1%)	Simon McVittie	27 (0.1%)
Javier Jardón	3 (1.1%)	Emmanuele Bassi	25 (0.1%)
Christian Hergert	3 (1.1%)	Robert Ancell	23 (0.1%)
coypu	2 (0.7%)	Marc-André Lureau	14 (0.1%)
Sebastian Geiger	2 (0.7%)	Jan de Groot	14 (0.1%)

Ignacio has been hard at work, helped by Ruslan and Fan, in making 2.48 the best GLib release ever in terms of supporting Windows — both for cross and native compilation, using autotools and the Microsoft Visual C compiler suite. If you can build an application for Windows as reliably as you can on Linux, it’s because of their work.

GTK+

For GTK+, on the other hand, the most active of the 81 contributors are:

Name	Per changeset	Name	Per changed lines
Matthias Clasen	1220 (43.7%)	Matthias Clasen	78960 (41.1%)
Benjamin Otte	472 (16.9%)	Benjamin Otte	35975 (18.7%)
Lapo Calamandrei	203 (7.3%)	Lapo Calamandrei	35352 (18.4%)
Cosimo Cecchi	167 (6.0%)	Cosimo Cecchi	10408 (5.4%)
Carlos Garnacho	147 (5.3%)	Jakub Steiner	6927 (3.6%)
Timm Bäder	107 (3.8%)	Carlos Garnacho	5334 (2.8%)
Emmanuele Bassi	41 (1.5%)	Alexander Larsson	3128 (1.6%)
Paolo Borelli	39 (1.4%)	Chun-wei Fan	2394 (1.2%)
Ruslan Izhbulatov	29 (1.0%)	Paolo Borelli	1771 (0.9%)
Carlos Soriano	28 (1.0%)	Ruslan Izhbulatov	1635 (0.9%)
Jakub Steiner	26 (0.9%)	Timm Bäder	1326 (0.7%)
Olivier Fourdan	26 (0.9%)	Takao Fujiwara	1269 (0.7%)
Jonas Ådahl	23 (0.8%)	Jonas Ådahl	1243 (0.6%)
Chun-wei Fan	22 (0.8%)	Emmanuele Bassi	885 (0.5%)
Piotr Drąg	18 (0.6%)	Olivier Fourdan	646 (0.3%)
Ray Strode	18 (0.6%)	Ray Strode	570 (0.3%)
Ignacio Casal Quinteiro	16 (0.6%)	Sébastien Wilmet	494 (0.3%)
William Hua	16 (0.6%)	Carlos Soriano	427 (0.2%)
Alexander Larsson	14 (0.5%)	Ignacio Casal Quinteiro	333 (0.2%)
Christoph Reiter	10 (0.4%)	William Hua	321 (0.2%)

Benjamin has worked on the new CSS gadget internal API; Matthias, Cosimo, and Timm have worked on porting existing widgets to it, in order to validate the API. Lapo and Jakub have worked on updating Adwaita and the other in tree themes to the new style declarations.

Carlos Soriano has worked on the widgets shared between the file chooser dialog and Nautilus.

Carlos Garnacho has worked on the input layer in GDK, in order to make it behave correctly under the new world order of Wayland; and speaking of Wayland, Carlos, Jonas, and Olivier have worked really hard to implement all the missing features in the Wayland backend, as well as the fallout of the Wayland switch when it comes to window sizing and positioning.

Affiliation

GLib

Affiliation	Per changeset	Affiliation	Per lines	Affiliation	Per contributor (total 55)
(Unknown)	136 (50.2%)	(Unknown)	10942 (50.9%)	(Unknown)	35 (60.3%)
Collabora	49 (18.1%)	Collabora	5491 (25.6%)	Red Hat	9 (15.5%)
Canonical	41 (15.1%)	Red Hat	3398 (15.8%)	Canonical	5 (8.6%)
Red Hat	36 (13.3%)	Canonical	1612 (7.5%)	Collabora	4 (6.9%)
Endless	6 (2.2%)	Endless	34 (0.2%)	Endless	2 (3.4%)
Centricular	1 (0.4%)	Centricular	4 (0.0%)	Centricular	1 (1.7%)
Intel	1 (0.4%)	Intel	2 (0.0%)	Intel	1 (1.7%)
Novell	1 (0.4%)	Novell	1 (0.0%)	Novell	1 (1.7%)

As usual, GLib is a little bit more diverse, in terms of employers, because of its versatility and use in various platforms.

GTK+

Affiliation	Per changeset	Affiliation	Per lines	Affiliation	Per contributor (total 81)
Red Hat	1940 (69.5%)	Red Hat	131833 (68.7%)	(Unknown )	63 (75.9%)
(Unknown)	796 (28.5%)	(Unknown)	59204 (30.8%)	Red Hat	15 (18.1%)
Endless	41 (1.5%)	Endless	885 (0.5%)	Canonical	4 (4.8%)
Canonical	13 (0.5%)	Canonical	104 (0.1%)	Endless	1 (1.2%)

Not many changes in these tables, but if your company uses the GNOME core platform and you wish to have a voice in where the platform goes, you should really consider contributing employee time to work upstream.

It is also very important to note that, while Red Hat still retains the majority of commits, the vast majority of committers are unaffiliated.

Methodology

The command line I used for gitdm is:

git log \
 --numstat \
 -M $START..$END | \
 gitdm -r '.*(?<!po)$' -l 20 -u -n

For GLib, I started from commit 37fcab17 which contains the version bump to 2.47, and ended on the 2.48.0 tag.

For GTK+, I started from commit 2f0d4b68 which contains the first new API of the 3.19 cycle and precedes the version bump, and ended on the 3.20.0 tag.

The only changes to the gitdm stock configuration are the addition of a couple of email/name/employer association; I can publish them on request.

Who wrote GTK+ (Reprise)

2015-09-16T00:00:00+01:00

As I’ve been asked by different people about data from older releases of GTK+, after the previous article on Who Wrote GTK+ 3.18, I ran the git-dm script on every release and generated some more data:

Release	Lines added	Lines removed	Delta	Changesets	Contributors
2.0¹	666495	345348	321147	2503	106
2.2	301943	227762	74181	1026	89
2.4	601707	116402	485305	2118	109
2.6	181478	88050	93428	1421	101
2.8	93734	47609	46125	1155	86
2.10	215734	54757	160977	1614	110
2.12	232831	43172	189659	1966	148
2.14	215151	102888	112263	1952	140
2.16	71335	23272	48063	929	118
2.18	52228	23490	28738	1079	90
2.20	80397	104504	-24107	761	82
2.22	51115	71439	-20324	438	70
2.24	4984	2168	2816	184	37
3.0¹	354665	580207	-225542	4792	115
3.2	227778	168616	59162	2435	98
3.4	126934	83313	43621	2201	84
3.6	206620	34965	171655	1011	89
3.8	84693	34826	49867	1105	90
3.10	143711	204684	-60973	1722	111
3.12	86342	54037	32305	1453	92
3.14	130387	144926	-14539	2553	84
3.16	80321	37037	43284	1725	94
3.18*	78997	54614	24383	1638	83

Here you can see the history of the GTK releases, since 2.0.

These numbers are to be taken with a truckload of salt, especially the ones from the 2.x era. During the early 2.x cycle, releases did not follow the GNOME timed release schedule; instead, they were done whenever needed:

Release	Date
2.0	March 2002
2.2	December 2002
2.4	March 2004
2.6	December 2004
2.8	August 2005
2.10	July 2006
2.12	September 2007
2.14	September 2008
2.16	March 2009
2.18	September 2009
2.20	March 2010
2.22	September 2010
2.24	January 2011

Starting with 2.14, we settled to the same cycle as GNOME, as it made releasing GNOME and packaging GTK+ on your favourite distribution a lot easier.

This disparity in the length of the development cycles explains why the 2.12 and 2.14 cycles, which lasted a year, represent an anomaly in terms of contributors (148 and 140, respectively) and in terms of absolute lines changed.

The reduced activity between 2.20 and 2.24.0 is easily attributable to the fact that people were working hard on the 2.90 branch that would become 3.0.

In general, once you adjust by release time, it’s easy to see that the number of contributors is pretty much stable at around 90:

The average is 94.5, which means we have an hobbit somewhere in the commit log

Another interesting data point would be to look at the ecosystem of companies spawned around GTK+ and GNOME, and how it has changed over the years — but that’s part of a larger discussion that would probably take more than a couple of blog posts to unpack.

I guess the larger point is that GTK+ is definitely not dying; it’s pretty much being worked on by the same amount of people — which includes long timers as well as newcomers — as it was during the 2.x cycle.

Both 2.0 and 3.0 are not wholly accurate; I used, as a starting point for the changeset period, the previous released branch point; for GTK+ 2.0, I started from the GTK_1_3_1 tag, whereas for GTK+ 3.0 I used the 2.90.0 tag. There are commits preceding both tags, but not enough to skew the results. ↩↩

Who wrote GTK+ 3.18

2015-09-15T00:00:00+01:00

It’s common “knowledge” in the Internet Peanut Gallery that GTK+ is “dead” or “dying” — I assume in the same sense that NetCraft certified that BSD is dead. It’d be (and, in point of fact, it is) easy to dismiss these rumors; it’s not like they come with actual numbers and trends, because the gods of old never mentioned the requirement for comments on the Internet to be cogent, let alone factually true, when they laid down the various RFCs.

On the other hand, not having an actual answer is a bit of a self-serving argument on the side of people, like me, that actually contribute to GTK+ and to the GNOME core platform; it allows a bit of leeway when we ask for help, contributions, or simply for decent bug reports — “we don’t have enough resources, so if you want your pet bug fixed, or feature implemented, you’ll have to help us help you”. Not that having actual numbers would change that; after all, resources are not infinite, and there are a ton of bugs that need to be fixed — including ones that require a time machine.

As much as I wanted to dispel the rumors about the impending death of GTK+, my goal was definitely to provide numbers on how much gets done every six months in the core GNOME platform, thus honoring the people that work hard on it. Having numbers also allows us to quantify what kind of help we need, so that, the next time somebody complains that we don’t work hard enough on fixing ALL the bugs, at least we’ll have a handy way to point out how much we work already.

To gather the data, I’ve used the most excellent git-dm tool that Jonathan Corbet wrote for the “Who wrote the Linux kernel” columns for LWN.

Activity

To provide a baseline, this is how the development activity looked like during the GNOME 3.14 and 3.16 cycles, i.e. during the past year:

	Version	Lines added	Lines removed	Delta	Contributors
GLib	2.42	17195	9934	7261	61
GLib	2.44	12504	2240	10264	49
GTK+	3.14	130387	144945	-14558	84
GTK+	3.16	80321	37043	43278	94

Note: the numbers do not include the translation work; as much as translations are an important part of our stack, they tend to skew any statistic, given the sheer size of lines touched.

The 3.14 cycle is an outlier for GTK+ because of the move of Adwaita in tree, as well as the GTK+ Inspector.

For the 3.18 cycle, the numbers up to September 15th are:

	Version	Lines added	Lines removed	Delta	Contributors
GLib	2.46	19763	12437	7326	50
GTK+	3.18	78676	54508	24168	83

As you can see, the numbers are mostly stable, in terms for code changes and number of developers.

Contributors

Of the 50 developers that contributed the 355 changesets of GLib during the 3.18 cycle, the most active are:

Name	Per changeset	Name	Per changed lines
Matthias Clasen	89 (25.9%)	Руслан Ижбулатов	7337 (27.6%)
Philip Withnall	50 (14.5%)	Ryan Lortie	5709 (21.5%)
Ryan Lortie	31 (9.0%)	Chun-wei Fan	3426 (12.9%)
Dan Winship	29 (8.4%)	Matthias Clasen	2881 (10.8%)
Simon McVittie	19 (5.5%)	Philip Withnall	1729 (6.5%)
Chun-wei Fan	14 (4.1%)	Dan Winship	1590 (6.0%)
Руслан Ижбулатов	11 (3.2%)	Simon McVittie	867 (3.3%)
Mikhail Zabaluev	8 (2.3%)	Alexander Larsson	648 (2.4%)
Ting-Wei Lan	8 (2.3%)	Paolo Borelli	588 (2.2%)
Garrett Regier	6 (1.7%)	Patrick Griffis	358 (1.3%)
Alexander Larsson	5 (1.5%)	Janusz Lewandowski	313 (1.2%)
Michael Catanzaro	5 (1.5%)	Mikhail Zabaluev	256 (1.0%)
Emmanuele Bassi	5 (1.5%)	Iain Lane	130 (0.5%)
Christophe Fergeau	5 (1.5%)	Garrett Regier	128 (0.5%)
Paolo Borelli	5 (1.5%)	Michael Catanzaro	106 (0.4%)
Piotr Drąg	4 (1.2%)	Richard Hughes	97 (0.4%)
Kalev Lember	4 (1.2%)	Emmanuele Bassi	70 (0.3%)
Iain Lane	4 (1.2%)	Xavier Claessens	48 (0.2%)
Patrick Griffis	4 (1.2%)	Ross Lagerwall	35 (0.1%)
Rico Tzschichholz	3 (0.9%)	Ting-Wei Lan	32 (0.1%)

Руслан Ижбулатов has been working on the Windows support, ensuring that the library and test suites work correctly there. Chun-wei Fan has been fixing the project files for building GLib (and GTK+, as well as a lot of libraries in the GNOME stack) with Microsoft Visual Studio and the Microsoft Visual C Compiler. Philip Withnall has been hard at work on the API reference and GObject tutorial, incorporating the feedback he got from clients at Collabora. Dan Winship and Michael Catanzaro have been working on the certificate API inside GIO, even though the bulk of the work has been going on inside the external glib-networking module. Simon McVittie has been working on GDBus; on the testing API; and has been incorporating patches coming from the Debian project.

For GTK+, on the other hand, the most active of the 83 contributors are:

Name	Per changeset	Name	Per changed lines
Matthias Clasen	811 (49.8%)	Matthias Clasen	37393 (37.3%)
Benjamin Otte	184 (11.3%)	Chun-wei Fan	22644 (22.6%)
Carlos Garnacho	107 (6.6%)	Benjamin Otte	10991 (11.0%)
Cosimo Cecchi	40 (2.5%)	Jakub Steiner	4762 (4.7%)
Jakub Steiner	37 (2.3%)	Georges Basile Stavracas Neto	3879 (3.9%)
Lapo Calamandrei	35 (2.1%)	Carlos Soriano	3827 (3.8%)
Emmanuele Bassi	33 (2.0%)	Lapo Calamandrei	3208 (3.2%)
Carlos Soriano	30 (1.8%)	Carlos Garnacho	2690 (2.7%)
Timm Bäder	29 (1.8%)	Руслан Ижбулатов	1480 (1.5%)
Chun-wei Fan	24 (1.5%)	Alexander Larsson	1001 (1.0%)
William Hua	24 (1.5%)	William Hua	947 (0.9%)
Alexander Larsson	23 (1.4%)	Cosimo Cecchi	704 (0.7%)
Georges Basile Stavracas Neto	23 (1.4%)	Paolo Borelli	671 (0.7%)
Jonas Ådahl	19 (1.2%)	Jasper St. Pierre	627 (0.6%)
Christian Hergert	17 (1.0%)	Christian Hergert	592 (0.6%)
Piotr Drąg	17 (1.0%)	Sebastien Lafargue	570 (0.6%)
Paolo Borelli	17 (1.0%)	Emmanuele Bassi	556 (0.6%)
Christoph Reiter	14 (0.9%)	Jonas Ådahl	543 (0.5%)
Руслан Ижбулатов	13 (0.8%)	Christoph Reiter	488 (0.5%)
Jasper St. Pierre	11 (0.7%)	Ryan Lortie	424 (0.4%)

While Benjamin is hard at work at improving the correctness and performance of the style machinery inside GTK+, Jakub and Lapo are constantly trying to find ways to make Adwaita and the High Constrast themes push the boundaries of the same style machinery. Carlos Soriano and Georges Basile Stavracas Neto have been working on the components of the file selection dialog following the new designs from Allan Day, for the Google Summer of Code; the code is going to be shared between GTK+ and Nautilus, to improve consistency between Nautilus and the GtkFileChooser widget, and keep bugs to a minimum. Carlos Garnacho has worked on input — mostly the support for touchpad on Wayland. Also on Wayland, Jonas Ådahl has been working on bug fixing and feature parity in GDK between Wayland and X11.

From a company perspective, Red Hat still dominates, as it employs many of the more prolific contributors; nevertheless, it’s important to note that a larger number of developers are unaffiliated, or contribute to GLib and GTK+ in their own time:

GLib

Affiliation	Per changeset	Affiliation	Per lines	Affiliation	Per contributor (total 52)
Red Hat	138 (40.1%)	(Unknown)	12892 (48.5%)	(Unknown)	32 (61.5%)
(Unknown)	95 (27.6%)	Canonical	5794 (21.8%)	Red Hat	10 (19.2%)
Collabora	65 (18.9%)	Red Hat	5222 (19.6%)	Canonical	3 (5.8%)
Canonical	37 (10.8%)	Collabora	2574 (9.7%)	Collabora	2 (3.8%)
Endless	6 (1.7%)	Endless	72 (0.3%)	Endless	2 (3.8%)
Centricular	2 (0.6%)	Centricular	28 (0.1%)	Centricular	2 (3.8%)

GTK+

Affiliation	Per changeset	Affiliation	Per lines	Affiliation	Per contributor (total 85)
Red Hat	1230 (75.5%)	Red Hat	61725 (61.6%)	(Unknown)	58 (68.2%)
(Unknown)	340 (20.9%)	(Unknown)	36845 (36.7%)	Red Hat	19 (22.4%)
Endless	43 (2.6%)	Endless	1181 (1.2%)	Canonical	3 (3.5%)
Canonical	13 (0.8%)	Canonical	491 (0.5%)	Endless	2 (2.4%)
Collabora	2 (0.1%)	Collabora	19 (0.0%)	Collabora	2 (2.4%)
Intel	1 (0.1%)	Intel	2 (0.0%)	Intel	1 (1.2%)

Conclusions

One of the most obvious conclusions that I can draw from these numbers is that GLib and GTK+ are definitely capable of retaining existing contributors — you just need to look at the names in the top committers and check how many GUADECs they have attended; what’s less obvious is the capacity of acquiring and retaining new contributors. For the latter, the Summer of Code and Outreachy programs are definitely a great resource. Carlos and Georges have been working their way down the stack at an impressive speed, and are now responsible for core functionality.

In terms of contributions, I think the code base has long since reached a point where it cannot be increased without also increasing the number of stable contributors. This is not a bad thing, per se; GLib and GTK+ are not the Linux kernel; we cannot add widgets like the kernel adds drivers, or file systems. It is mostly clear, though, that new functionality must come with new people that take responsibility for it, or at the expense of deprecating old API.

PyClutter Reborn

2015-05-29T12:00:00+01:00

Back when I started hacking on Clutter, the Python bindings were the first thing that I was tasked to write. In order to make them happen, I had to understand the intent of the library, and fix up the API to be binding-friendly. It was another time — a time before introspection made possible to have run-time bindings that automatically acquired support for new API once the underlying shared library changed. I had to learn the CPython API; the code generation utilities that turned C header files into defs files, and defs files into C code; I also had to fix those utilities, whenever they lied about what they did, or whenever they were pretty much ad hoc code supporting GTK+ quirks; and, finally, I had to understand the build system used by PyGTK that ate all this stuff on one end, blended it all, and spat out a Python C module on the other.

For a long while, the Python bindings for Clutter have been based on that particular brand on insanity; people were able to switch to the introspection-based ones and drop PyClutter entirely, but the API was not as nice to use, and you’d be missing out on some niceties provided by both Python and Clutter.

In 2011, Bastian Winkler started porting the whole infrastructure to the overrides mechanism provided by PyGObject; instead of just loading the Clutter introspection data, you’d load a pure Python module that wrapped specific bits of the exposed API, and made it nicer to use.

For various reasons — mostly lack of time on my side and lack of testing — that work has been bitrotting in a branch of the Git repository. Well, no more. The master branch of the PyClutter repository is now providing introspection overrides similar to the ones for GTK+. I also added a bunch of examples, ported from their C equivalent, to show the idiomatic use of Clutter in a Python context.

I’ll probably do a release, but I’d be happy if somebody wanted to pick up the bindings and run with them — I’m not much of a Python programmer myself.

Resources

PyClutter Git repository on git.gnome.org
PyClutter examples

halting problem - gnome

Let’s talk about Moonforge

What is Moonforge

Why?

A little detour: Poky

Another little detour: Kas

What Moonforge provides

Trying Moonforge out

Governance in GNOME

How do things happen in GNOME?

GNOME’s original sin

Herding cats and vectors sum

On maintainers

Painting a target on your back

Burnout and you

Governance

Examples

Python

Rust

Fedora

GNOME

Goals

Diffusing responsibilities

Empowering the community

Empowering individuals

Actual decision making process

Proposed GNOME Changes✝

PGCs

Editorial team

Steering group

Is this enough?

Whither maintainers

Embracing special interest groups

What happens if I say no?

Civil service

To sum up

The next steps

Introspection’s edge

The Mirror

The Mirror

Things that were

Things that are

Things that yet may be

A new, old base type

Optional functionality

Typed containers and life time management

Specialised types

Tuple types

Sum types

Consolidating GLib and GType

Properties

Signals

Extending the type system

Introspection

There and back again

Configuring portals

Writing Bindable API, 2023 Edition

Structures with custom memory management

Addendum: object types

Functionality only accessible through a C macro

Addendum: inline functions

Direct C structure access for objects

va_list

Multiple out parameters

Addendum: inout arguments

Addendum: GValue

Arrays

Addendum: strings and byte arrays

Addendum: GArray, GPtrArray, GByteArray

Addendum: GHashTable

Addendum: GList, GSList, GQueue

Callbacks

Addendum: the closure annotation

On PyGObject

Case study: typed instances

Case study: constructing and disposing

Case study: documentation, examples, and tutorials

What can you do?

What happens if nobody does anything?

Amberol

`va_list`

Addendum: `inout` arguments

Addendum: `GValue`

Addendum: `GArray`, `GPtrArray`, `GByteArray`

Addendum: `GHashTable`

Addendum: `GList`, `GSList`, `GQueue`

Addendum: the `closure` annotation