Maarten Balliauw {blog}

ASP.NET MVC, Microsoft Azure, PHP, web development ...

NAVIGATION - SEARCH

How we built TwitterMatic.net - Part 3: Store data in the cloud

TwitterMatic - Schedule your Twitter updates “After setting up his workplace, knight Maarten The Brave Coffeedrinker thought of something else: if a farmer wants to keep a lot of hay, he needs a barn, right? Since the cloudy application would also need to keep things that can be used by the digital villagers, our knight needs a barn in the clouds. Looking at the azure sky, an idea popped into the knight’s head: why not use Windows Azure storage service? It’s a barn that’s always there, a barn that can catch fire and will still have its stored items located in a second barn (and a third). Knight Maarten The Brave Coffeedrinker jumped on his horse and went on a quest, a quest in the clouds.

This post is part of a series on how we built TwitterMatic.net. Other parts:

kick it on DotNetKicks.com

Store data in the cloud

Windows Azure offers 3 types of cloud storage: blobs, tables and queues. Blob Storage stores sets of binary data, organized in containers of your storage account. Table Storage offers structured storage in the form of tables. The Queue service stores an unlimited number of messages, each of which can be up to 8 KB in size.

Let’s look back at the TwitterMatic architecture:

“The worker role will monitor the table storage for scheduled Tweets. If it’s time to send them, the Tweet will be added to a queue. This queue is then processed by another thread in the worker role, which will publish the Tweet to Twitter. ”

This means we’ll be using two out of three storage types: Table Storage and Queue Storage. Problem: these services are offered as a RESTful service, somewhere in the cloud. Solution to that: use the StorageClient project located in the Windows Azure SDK’s samples directory!

The StorageClient project contains .NET API’s that work with the blob, table and queue storage services provided by Windows Azure. For Table Storage, StorageClient provides an extension on top of Astoria (ADO.NET Data Services Framework, but I still say Astoria because it’s easier to type and say…). This makes it easier for you as a developer to use existing knowledge, from LINQ to SQL or Entity Framework or Astoria, to develop Windows Azure applications.

Setting up StorageClient

Add a reference to the StorageClient project to your WebRole project. Next, add some settings to the ServiceConfiguration.cscfg file:

[code:xml]

<?xml version="1.0"?>
<ServiceConfiguration serviceName="TwitterMatic" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration">
  <Role name="WebRole">
    <Instances count="1"/>
    <ConfigurationSettings>
      <Setting name="AccountName" value="twittermatic"/>
      <Setting name="AccountSharedKey" value=”..."/>
      <Setting name="BlobStorageEndpoint" value="http://blob.core.windows.net"/>
      <Setting name="QueueStorageEndpoint" value = "http://queue.core.windows.net"/>
      <Setting name="TableStorageEndpoint" value="http://table.core.windows.net"/>
      <Setting name="allowInsecureRemoteEndpoints" value="true"/>
    </ConfigurationSettings>
  </Role>
  <Role name="WorkerRole">
    <Instances count="1"/>
    <ConfigurationSettings>
      <Setting name="AccountName" value="twittermatic"/>
      <Setting name="AccountSharedKey" value=”..."/>
      <Setting name="BlobStorageEndpoint" value="http://blob.core.windows.net"/>
      <Setting name="QueueStorageEndpoint" value = "http://queue.core.windows.net"/>
      <Setting name="TableStorageEndpoint" value="http://table.core.windows.net"/>
      <Setting name="allowInsecureRemoteEndpoints" value="true"/>
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

[/code]

This way, both the web and worker role know where to find their data (URI) and how to authenticate (account name and shared key).

Working with tables

We’ll only be using one domain class in our entire project: TimedTweet. This class represents a scheduled Twitter update, containing information required to schedule the update. Here’s a list of properties:

  • Token: A token used to authenticate against Twitter
  • TokenSecret: A second token used to authenticate against Twitter
  • ScreenName: Twitter screen name of the user.
  • Status: The message to publish on Twitter.
  • SendOn: Time to send the message.
  • SentOn: Time the message was sent.
  • SendStatus: A status message (pending, in progress, published, …)
  • RetriesLeft: How many retries left before giving up on the Twitter update.
  • Archived: Yes/no if the message is archived.

Here’s the code:

[code:c#]

public class TimedTweet : TableStorageEntity, IComparable
{
    public string Token { get; set; }
    public string TokenSecret { get; set; }
    public string ScreenName { get; set; }
    public string Status { get; set; }
    public DateTime SendOn { get; set; }
    public DateTime SentOn { get; set; }
    public string SendStatus { get; set; }
    public int RetriesLeft { get; set; }
    public bool Archived { get; set; }

    public TimedTweet()
        : base()
    {
        SendOn = DateTime.Now.ToUniversalTime();
        Timestamp = DateTime.Now;
        RowKey = Guid.NewGuid().ToString();
        SendStatus = "Scheduled";
        RetriesLeft = 3;
    }

    public TimedTweet(string partitionKey, string rowKey)
        : base(partitionKey, rowKey)
    {
        SendOn = DateTime.Now.ToUniversalTime();
        SendStatus = "Scheduled";
        RetriesLeft = 3;
    }

    public int CompareTo(object obj)
    {
        TimedTweet target = obj as TimedTweet;
        if (target != null) {
            return this.SendOn.CompareTo(target.SendOn);
        }
        return 0;
    }
}

[/code]

Note that our TimedTweet is inheriting TableStorageEntity. This class provides some base functionality for Windows Azure Table Storage.

We’ll also need to work with this class against table Storage. For that, we can use TableStorage and TableStorageDataServiceContext class, like this:

[code:c#]

public List<TimedTweet> RetrieveAllForUser(string screenName) {
    StorageAccountInfo info = StorageAccountInfo.GetDefaultTableStorageAccountFromConfiguration(true);
    TableStorage storage = TableStorage.Create(info);

    storage.TryCreateTable("TimedTweet");

    TableStorageDataServiceContext svc = storage.GetDataServiceContext();
    svc.IgnoreMissingProperties = true;

    List<TimedTweet> result = svc.CreateQuery<TimedTweet>("TimedTweet").Where(t => t.ScreenName ==     screenName).ToList();
    foreach (var item in result)
    {
        svc.Detach(item);
    }
    return result;
}

[/code]

Using this, we can build a repository class based on ITimedtweetRepository and implemented against Table Storage:

[code:c#]

public interface ITimedTweetRepository
{
    void Delete(string screenName, TimedTweet tweet);
    void Archive(string screenName, TimedTweet tweet);
    void Insert(string screenName, TimedTweet tweet);
    void Update(TimedTweet tweet);
    List<TimedTweet> RetrieveAll(string screenName);
    List<TimedTweet> RetrieveDue(DateTime dueDate);
    TimedTweet RetrieveById(string screenName, string id);
}

[/code]

Working with queues

Queues are slightly easier to work with: you can enqueue messages and dequeue messages, and that’s about it. Here’s an example snippet:

[code:c#]

public void EnqueueMessage(string message) {
    StorageAccountInfo info = StorageAccountInfo.GetDefaultTableStorageAccountFromConfiguration(true);

    QueueStorage queueStorage = QueueStorage.Create(info);
    MessageQueue updateQueue = queueStorage.GetQueue("updatequeue");
    if (!updateQueue.DoesQueueExist())
        updateQueue.CreateQueue();

    updateQueue.PutMessage(new Message(message));
}

Conclusion

[/code]

We now know how to use StorageClient and have a location in the cloud to store our data.

In the next part of this series, we’ll have a look at how we can leverage Twitter’s OAuth authentication mechanism in our own TwiterMatic application and make use of some other utilities packed in the Windows Azure SDK.

kick it on DotNetKicks.com

How we built TwitterMatic.net - Part 2: Creating an Azure project

TwitterMatic - Schedule your Twitter updates “Knight Maarten The Brave Coffeedrinker was about to start working on his TwitterMatic application, named after the great god of social networking, Twitter. Before he could start working, he first needed the right tools. He downloaded the Windows Azure SDK, a set of tools recommended by the smith (or was it the carpenter?) of the digital village. Our knight’s work shack was soon ready to start working. The table on which the application would be crafted, was still empty. Time for action, the knight thought. And he started working.”

This post is part of a series on how we built TwitterMatic.net. Other parts:

kick it on DotNetKicks.com

Creating an Azure project

Here we go. After installing the Windows Azure SDK, start a new project: File > New > Project... Next, add a “Web And Worker Cloud Service”, located under Visual C# (or VB...) > Cloud Service > Web And Worker Cloud Service.

The project now contains three projects:

TwitterMatic Visual Studio Solution

The web role project should be able to run ASP.NET, but not yet ASP.NET MVC. Add /Views, /Controllers, Global.asax.cs, references to System.Web.Mvc, System.Web.Routing, … to make the web role an ASP.NET MVC project. This should work out fine, except for the tooling in Visual Studio. To enable ASP.NET MVC tooling, open the TwitterMatic_WebRole.csproj file using Notepad and add the following: (copy-paste: {603c0e0b-db56-11dc-be95-000d561079b0})

Enable ASP.NET MVC tooling in Windows Azure project

Visual Studio will prompt to reload the project, allow this by clicking the "Reload" button.

(Note: we could have also created a web role from an ASP.NET MVC project, check my previous series on Azure to see how to do this.)

Sightseeing

True, I ran over the previous stuff a bit fast, as it is just plumbing ASP.NET MVC in a Windows Azure Web Role. I have written about this in more detail before, check my previous series on Azure. There are some other things I would like to show you though.

The startup project in a Windows Azure solution contains some strange stuff:

TwitterMatic startup project

This project contains 3 types of things: the roles in the application, a configuration file and a file describing the required configuration entries. The “Roles” folder contains a reference to the projects that perform the web role on one side, the worker role on the other side. ServiceConfiguration.cscfg contains all configuration entries for the web and worker roles, such as the number of instances (yes, you can add more servers simply by putting a higher integer in there!) and other application settings such as storage account info.

ServiceDefinition.csdef is basically a copy of the other config file, but without the values. If you are wondering why: if someone screws with ServiceConfiguration.cscfg when deploying the application to Windows Azure, the deployment interface will know that there are settings missing or wrong. Perfect if someone else will be doing deployment to production!

Conclusion

We now have a full Windows Azure solution, with ASP.NET MVC enabled! Time to grab a prefabricated CSS theme, add a logo, … Drop-in ASP.NET MVC themes can always be found at http://www.asp.net/mvc/gallery. The theme we used should be there as well (thanks for that, theme author!).

In the next part of this series, we’ll have a look at where and how we can store our data.

kick it on DotNetKicks.com

How we built TwitterMatic.net - Part 1: Introduction

TwitterMatic “Once upon a time, Microsoft started a Windows Azure developing contest named new CloudApp();. While it first was only available for US candidates, the contest was opened for international submissions too. Knight Maarten The Brave Coffeedrinker and his fellow knightsmen at RealDolmen decided to submit a small sample application that could be hosted in an unknown environment, known by the digital villagers as “the cloud”. The application was called TwitterMatic, named after the great god of social networking, Twitter. It would allow digital villagers to tell the latest stories, even when they were asleep or busy working.”

There, a nice fairy tale :-) It should describe the subject of a blog post series that I am starting around the techncal background of TwitterMatic, our contest entry for the new CloudApp(); contest. Now don't forget to vote for us between 10 July and 20 July!

Some usage scenario’s for TwitterMatic:

  • Inform your followers about interesting links at certain times of day
  • Stay present on Twitter during your vacation
  • Maintain presence in your activity stream, even when you are not available
  • Never forget a fellow Twitterer's birthday: schedule it!
  • Trick your boss: of course you are Tweeting you're leaving the office at 8 PM!

Perfect excuses to build our application for the clouds. Now for something more interesting: the technical side!

If you are impatient and immediately want the source code for TwitterMatic, check http://twittermatic.codeplex.com.

kick it on DotNetKicks.com

TwitterMatic architectural overview

Since we’re building a demo application, we thought: why not make use of as much features as possible which Windows Azure has to offer? We’re talking web role, worker role, table storage and queue storage here!

  • The web role will be an application built in ASP.NET MVC, allowing you to schedule new Tweets and view archived scheduled Tweets.
  • The worker role will monitor the table storage for scheduled Tweets. If it’s time to send them, the Tweet will be added to a queue. This queue is then processed by another thread in the worker role, which will publish the Tweet to Twitter.
  • We’ll be using OAuth, delegating authentication for TwitterMatic to Twitter itself. This makes it easier for us: no need to store credentials, no need to maintain a user database, …
  • The web role will perform validation of the domain using data annotations. More on this in one of the next parts.

For people who like images, here’s an architecture image:

TwitterMatic Architecture

What’s next?

RealDolmen Windows Azure The next parts of this series around Windows Azure will be focused on the following topics:

Stay tuned during the coming weeks! And don’t forget to start scheduling Tweets using TwitterMatic.

kick it on DotNetKicks.com