Network Programming in .NET

I wanted to do something quite simple, plot tweets live on a map, that is, as people posted tweets live on twitter, drop a pin on the map, and display the tweet in a message bubble. I’ve done that, and popped it up on a website  “http://geotweet.webtropy.com/” – It takes a few seconds to start up, but goes quickly then.

I used the twitter stream API, which is an unusual API, since once you call it, it doesn’t close the connection, just keeps pumping out data.  I used the API call https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56 – now, this does require a twitter username and password, but not any fancy OAUTH authentication.

The bounding box -7,55,-6,56 is roughly the area shown in the screenshot above, but some tweets land outside of this area, you may need extra filtering if this is important to you.

I needed to write a bit of .NET that could extract one tweet at a time from the stream. Each tweet is a Json object, and they are delimited by new line chars.

public static string ReadLineFromUrl(string url)
{
string strData = “”;
var httprequest = (HttpWebRequest)WebRequest.Create(url);
httprequest.Credentials = new NetworkCredential(“your_twitter_name”, “your_twitter_password”);
var httpresponse = (HttpWebResponse)httprequest.GetResponse();
var responsestream = httpresponse.GetResponseStream();
StreamReader httpstream = new StreamReader(responsestream, Encoding.GetEncoding(“iso8859-1”));
strData = httpstream.ReadLine();
httprequest.Abort();
return strData;
}

Then, I created a page that calls this function, and writes it to screen

var strUrl = “https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56”;
var strResponse = FireHose.ReadLineFromUrl(strUrl);
Response.Write(strResponse);
Response.End();

You can modify the bounding box to meet your needs.

The source code for the web page is visible through view-source, so I won’t include it here. All I did was, initialise a google map. Once the map was ready call my AJAX function above.  extract either the geo or bounding_box object from it, and plot a marker on the map at that position. Once that is done, the Ajax is called again, and so the cycle continues.

***Update***

So, after a day of testing this new site, I suddenly got hit with a “402 client error” from twitter, which is funnily described as “Enhance your calm” on Twitter’s documentation. Basically, you’re not supposed to connect and disconnect rapidly to the stream API, you are supposed to keep connected to the stream, and only reconnect if you have to.

Fair enough, but it requires quite a redesign for the code.

What you need to do, is kick off a thread from Global.asax on Application_Start like so:

protected void Application_Start(object sender, EventArgs e)
{
FireHose hose = new FireHose();
hose.Url = “https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56”;
Thread t = new Thread(new ThreadStart(hose.ReadConstantlyFromUrl));
t.Start();
}

I had to modify the “FireHose” class to have a new method called ReadConstantlyFromUrl, which opens a connection, reads a line at a time for the stream, setting a public static variable, and only disconnecting on error, or if the end of the stream is reached. I also added a cool off period of a second, so that the reconnection attempts wouldn’t bunch up.

public void ReadConstantlyFromUrl()
{
for (; ; )
{
try
{
var httprequest = (HttpWebRequest)WebRequest.Create(this.Url);
httprequest.Credentials = new NetworkCredential(“twitter_account”, “your_password”);
var httpresponse = (HttpWebResponse)httprequest.GetResponse();
var responsestream = httpresponse.GetResponseStream();
StreamReader httpstream = new StreamReader(responsestream, Encoding.GetEncoding(“iso8859-1”));
for (; ; )
{
FireHose.Tweet = httpstream.ReadLine();
if (FireHose.Tweet == null) break;
}
httpstream.Close();
Thread.Sleep(1000); // Cool off period
}
catch (Exception ex)
{
string sSource = “GeoTweet”;
string sLog = “Application”;

if (!EventLog.SourceExists(sSource))
EventLog.CreateEventSource(sSource, sLog);

EventLog.WriteEntry(sSource, ex.ToString());
}
}
}

Then, of course, I had to change the Ajax handler to simply read the value of FireHose.Tweet

while (string.IsNullOrEmpty(FireHose.Tweet))
{
Thread.Sleep(100);
}
Response.Write(FireHose.Tweet);
FireHose.Tweet = “”;

The Thread.Sleep allows the Ajax handler wait until a new tweet has arrived before writing to the output stream. Do note, that if there were more than one client visiting the page at the same time, then it would be a first-come-gets-the-tweet. Which is not ideal for most situations, but ok for my test.

**Update**

Working on a solution for multi-client access, so that one client would not “steal” the tweets that should be visible to all clients, I decided to add an extra field in the Ajax called “IgnoreTweetIf”, this is set to the length of the tweet (the full data, not the text only), and the Ajax handler will not return a tweet if the length of the last tweet it has matches the length of the tweet that the client last received. On first call this “IgnoreTweetIf” is set to zero, so that any tweet is returned.

Therefore, my Ajax handler is modified as follows;

var strIgnoreTweetIf = Request.QueryString[“IgnoreTweetIf”];
var intIgnoreTweetIf = Convert.ToInt16(strIgnoreTweetIf);
while (string.IsNullOrEmpty(FireHose.Tweet) || intIgnoreTweetIf == FireHose.Tweet.Length)
{
Thread.Sleep(100);
}
Response.Write(FireHose.Tweet);

And the Ajax code is modified thus;

function ajaxLoop(ignoreTweetIf) {
$.get(“NextTweet.aspx?IgnoreTweetIf=” + ignoreTweetIf, function (data) {
handleTweet(data);
ajaxLoop(data.length);
});
}