How do you synchronize a million to-do lists?

Edit on GitHub

Not this question, but a similar one, has been asked by one of our customers. An interesting question, isn’t it? Wait. It gets more interesting. I’ll sketch a fake scenario that’s similar to our customer’s question. Imagine you are building mobile applications to manage a simple to-do list. This software is available on Android, iPhone, iPad, Windows Phone 7 and via a web browser. One day, the decision to share to-do lists has been made. Me and my wife should be able to share one to-do list between us, having an up-to-date version of the list on every device we grant access to this to-do list. Now imagine there are a million of those groups, where every partner in the sync relationship has the latest version of the list on his device. In often a disconnected world.

How would you solve this?

My take: Windows Azure Service Bus Topics & Subscriptions

According to the Windows Azure Service Bus product description, it “implements a publish/subscribe pattern that delivers a highly scalable, flexible, and cost-effective way to publish messages from an application and deliver them to multiple subscribers.“ Interesting. I’m not going into the specifics of it (maybe in a next post), but the Windows Azure Service Bus gave me an idea: why not put all actions (add an item, complete a to-do) on a queue, tagged with the appropriate “group” metadata? Here’s the producer side:

Windows Azure Service Bus Topics

On the consumer side, our devices are listening as well. Every device creates its subscription on the service bus topic. These subscriptions are named per device and filtered on the SyncGroup metadata. The Windows Azure Service Bus will take care of duplicating messages to every subscription as well as keeping track of messages that have not been processed: if I’m offline, messages are queued. If I’m online, I receive messages targeted at my device:

Windows Azure Service Bus Subscritpions

The only limitation to this is keeping the number of topics & subscriptions below the limits of Windows Azure Service Bus. But even then: if I just make sure every sync group is on the same bus, I can scale out over multiple service buses.

How would you solve the problem sketched? Comments are very welcomed!

This is an imported post. It was imported from my old blog using an automated tool and may contain formatting errors and/or broken images.

Leave a Comment


3 responses

  1. Avatar for Paul Stovell
    Paul Stovell January 6th, 2012

    Hi Maarten,

    1. What kinds of limits exist for the service bus? Is each subscription expensive?
    2. What would you when I reset my device and need my todo list back (having already read messages from the queue)?

    I&#39m guessing for 2 you&#39d need to persist TODO items somewhere and just use messages to sync?


  2. Avatar for maartenba
    maartenba January 10th, 2012

    1) You are billed for "connected time" (per hour). There are limits (I guess something like 20k subscriptions per bus) but you can partition over multiple buses.

    2) Yes, that&#39s correct. The plan is to subscribe some back-end services (filtered on SyncGroup = %) and store these items in a database, optionally shareded for scale.

  3. Avatar for Abid Rahman
    Abid Rahman January 10th, 2012

    Interesting. Would be nice to see a more detailed post. Thanks for the post.