halting problem :: Lazy loading using the GLib main loop

I wrote this page in 2006, back when I was starting out with complex GTK+ user interfaces, and I had to reply multiple times to questions on IRC and mailing lists that asked how to use threads to load content into a list store; this was my attempt at demonstrating that you don't really need threads at all, but you need to break down your work and parcellize it in discrete steps.

Sometimes you have a long list of simple stuff you want to feed into a GtkListStore in order to view it with a GtkTreeView. Soon, you realize that simply dogfeeding the data using a loop like this:

list-store-add-loop.c download
static GtkWidget *
load_content (GPtrArray *stuff)
{
  /* create the model */
  GtkListStore *store = gtk_list_store_new (3,
                                            G_TYPE_STRING, /* foo */
                                            G_TYPE_STRING, /* bar */
                                            G_TYPE_BOOLEAN /* baz */);

  /* create the view */
  GtkWidget *view = gtk_tree_view_new_with_model (GTK_TREE_MODEL (store));

  /* load each object data into a row of the model */
  for (int i = 0; i < stuff->len; i++)
    {
      YourObject *obj = stuff->pdata[i];
      GtkTreeIter iter;

      gtk_list_store_append (store, &iter);
      gtk_list_store_set (store, &iter,
                          COLUMN_FOO, your_object_get_foo (obj),
                          COLUMN_BAR, your_object_get_bar (obj),
                          COLUMN_BAZ, your_object_get_baz (obj),
                          -1);
    }

  /* the view owns the model now */
  g_object_unref (store);

  return view;
}

is really inefficient, as the GtkTreeView has to handle every row insertion.

Moving the assignment of the model after the loop makes things slightly better:

list-store-add-loop-fixed.c download
static GtkWidget *
load_content (GPtrArray *stuff)
{
  /* create the model */
  GtkListStore *store = gtk_list_store_new (3,
                                            G_TYPE_STRING, /* foo */
                                            G_TYPE_STRING, /* bar */
                                            G_TYPE_BOOLEAN /* baz */);

  /* load each object data into a row of the model */
  for (int i = 0; i < stuff->len; i++)
    {
      YourObject *obj = stuff->pdata[i];
      GtkTreeIter iter;

      gtk_list_store_append (store, &iter);
      gtk_list_store_set (store, &iter,
                          COLUMN_FOO, your_object_get_foo (obj),
                          COLUMN_BAR, your_object_get_bar (obj),
                          COLUMN_BAZ, your_object_get_baz (obj),
                          -1);
    }

  /* create the view */
  GtkWidget *view = gtk_tree_view_new_with_model (GTK_TREE_MODEL (store));

  /* the view owns the model now */
  g_object_unref (store);

  return view;
}

but the population of the GtkListStore might take quite some time nevertheless, and block your UI.

At this point, people will begin thinking about using a thread to do the list loading without interfering with the main application loop.

Threads are desiderable only if your application is already threaded - but introducing threads just to load stuff into a ListStore is simply overkill; also, making the threading work with the GTK is always a bit hairy, especially if you also want to update the GUI while loading. Instead, you should use the great main loop implementation that GLib has. All you need to do is set up a little finite state machine for loading stuff, use an idle function that gets called with low priority, and then feed the populated ListStore to the TreeView.

First step: set up the finite state machine. We use four states, two of which are the actual states for loading the items into the model and the model to the view:

list-store-add-idle.c [Lines 1-10] download
/* states in the FSM */
enum {
  STATE_STARTED,  /* start state */

  STATE_LOADING,  /* feeding the items to the model */
  STATE_COMPLETE, /* feeding the model to the view */

  STATE_FINISHED  /* finish state */
};

we also set up an opaque container for some data to be used in the idle callbacks:

list-store-add-idle.c [Lines 11-32] download
/* data to be passed to the idle handler */
typedef struct
{
  /* the current state */
  guint state;

  /* the idle handler id */
  guint load_id;

  /* the model */
  GtkListStore *list_store;

  /* the view */
  GtkWidget *tree_view;

  /* the items to be loaded */
  GPtrArray *items

  /* the currently loaded item */
  guint current_item;
} IdleData;

Then, we set up the idle functions; we use the g_idle_add_full() function because we will reach the STATE_FINISHED state in a clean up function, and assign the model to the view there:

list-store-add-idle.c [Lines 33-81] download
static void cleanup_load_items (gpointer data);

static void
load_content (GPtrArray *stuff)
{
  IdleData *data;

  data = g_new (IdleData, 1);

  data->items = g_ptr_array_ref (items);

  data->n_loaded = 0;

  /* create the model */
  data->list_store = gtk_list_store_new (3,
                                         G_TYPE_STRING, /* foo */
                                         G_TYPE_STRING, /* bar */
                                         G_TYPE_BOOLEAN /* baz */);
  /* create the view */
  data->tree_view = gtk_tree_view_new ();

  data->state = STATE_STARTED;
  data->load_id = g_idle_add_full (G_PRIORITY_HIGH_IDLE,
                                   load_items_idle,
                                   data,
                                   cleanup_load_items);

  return data->tree_view;
}

/* clean up function */
static void
cleanup_load_items (gpointer data_)
{
  IdleData *data = data_;

  /* make sure we are in the right state */
  g_assert (data->state == STATE_FINISHED);

  /* assign the model to the view */
  gtk_tree_view_set_model (GTK_TREE_VIEW (data->tree_view),
                           GTK_TREE_MODEL (data->list_store));

  /* the view now owns the model */
  g_object_unref (data->list_store);

  g_ptr_array_unref (data->items);
  g_free (data);
}

The load_items_idle() function will be called each time the main loop is idle, and will load the content of the items list one item at a time; as soon as the load_items_idle() has finished its run on the entire list, the cleanup_load_items() function will be called.

list-store-add-idle.c [Lines 83-130] download
/* the actual loading function */
static gboolean
load_items_idle (gpointer data_)
{
  IdleData *data = data_;
  YourObject *obj;
  GtkTreeIter iter;

  /* make sure we are in the right state */
  g_assert (data->state == STATE_STARTED || data->state == STATE_LOADING)

  /* no items left */
  if (data->current_item == data->items->len)
    {
      /* set the correct state */
      data->state = STATE_FINISHED;

      /* remove the source.
       *
       * this will cause cleanup_load_items() to be called
       */
      return G_SOURCE_REMOVE;
    }

  /* we are now loading */
  if (data->state == STATE_STARTED)
    data->state = STATE_LOADING;

  /* get the next item in the list */
  obj = data->items->pdata[data->current_item];
  g_assert (obj != NULL);

  gtk_list_store_append (data->list_store, &iter);
  gtk_list_store_set (data->list_store, &iter,
                      COLUMN_FOO, your_object_get_foo (obj),
                      COLUMN_BAR, your_object_get_bar (obj),
                      COLUMN_BAZ, your_object_get_baz (obj),
                      -1);

  /* next item */
  data->current_item += 1;

  /* we can also update the UI, for instance with a nice progress bar */
  update_progress (data->current_item, data->items->len);

  /* continue loading */
  return G_SOURCE_CONTINUE;
}

As you can see, this function appends one item at a time, using the IdleData structure to store the data between the runs.

This is it. Now the ListStore will get loaded lazily inside an idle callback that should not mess up with the responsiveness of your UI, without having to resort to threads. Another reason why GTK rocks!