Home > Uncategorized > Plot tweets on a map

Plot tweets on a map

tweetsOnMap

I wanted to do something quite simple, plot tweets live on a map, that is, as people posted tweets live on twitter, drop a pin on the map, and display the tweet in a message bubble. I’ve done that, and popped it up on a website  “http://geotweet.webtropy.com/” – It takes a few seconds to start up, but goes quickly then.

I used the twitter stream API, which is an unusual API, since once you call it, it doesn’t close the connection, just keeps pumping out data.  I used the API call https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56 – now, this does require a twitter username and password, but not any fancy OAUTH authentication.

The bounding box -7,55,-6,56 is roughly the area shown in the screenshot above, but some tweets land outside of this area, you may need extra filtering if this is important to you.

I needed to write a bit of .NET that could extract one tweet at a time from the stream. Each tweet is a Json object, and they are delimited by new line chars.

public static string ReadLineFromUrl(string url)
{
string strData = “”;
var httprequest = (HttpWebRequest)WebRequest.Create(url);
httprequest.Credentials = new NetworkCredential(“your_twitter_name”, “your_twitter_password”);
var httpresponse = (HttpWebResponse)httprequest.GetResponse();
var responsestream = httpresponse.GetResponseStream();
StreamReader httpstream = new StreamReader(responsestream, Encoding.GetEncoding(“iso8859-1”));
strData = httpstream.ReadLine();
httprequest.Abort();
return strData;
}

Then, I created a page that calls this function, and writes it to screen

var strUrl = “https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56”;
var strResponse = FireHose.ReadLineFromUrl(strUrl);
Response.Write(strResponse);
Response.End();

You can modify the bounding box to meet your needs.

The source code for the web page is visible through view-source, so I won’t include it here. All I did was, initialise a google map. Once the map was ready call my AJAX function above.  extract either the geo or bounding_box object from it, and plot a marker on the map at that position. Once that is done, the Ajax is called again, and so the cycle continues.

***Update***

So, after a day of testing this new site, I suddenly got hit with a “402 client error” from twitter, which is funnily described as “Enhance your calm” on Twitter’s documentation. Basically, you’re not supposed to connect and disconnect rapidly to the stream API, you are supposed to keep connected to the stream, and only reconnect if you have to.

Fair enough, but it requires quite a redesign for the code.

What you need to do, is kick off a thread from Global.asax on Application_Start like so:

protected void Application_Start(object sender, EventArgs e)
{
FireHose hose = new FireHose();
hose.Url = “https://stream.twitter.com/1/statuses/filter.json?locations=-7,55,-6,56”;
Thread t = new Thread(new ThreadStart(hose.ReadConstantlyFromUrl));
t.Start();
}

I had to modify the “FireHose” class to have a new method called ReadConstantlyFromUrl, which opens a connection, reads a line at a time for the stream, setting a public static variable, and only disconnecting on error, or if the end of the stream is reached. I also added a cool off period of a second, so that the reconnection attempts wouldn’t bunch up.

public void ReadConstantlyFromUrl()
{
for (; ; )
{
try
{
var httprequest = (HttpWebRequest)WebRequest.Create(this.Url);
httprequest.Credentials = new NetworkCredential(“twitter_account”, “your_password”);
var httpresponse = (HttpWebResponse)httprequest.GetResponse();
var responsestream = httpresponse.GetResponseStream();
StreamReader httpstream = new StreamReader(responsestream, Encoding.GetEncoding(“iso8859-1”));
for (; ; )
{
FireHose.Tweet = httpstream.ReadLine();
if (FireHose.Tweet == null) break;
}
httpstream.Close();
Thread.Sleep(1000); // Cool off period
}
catch (Exception ex)
{
string sSource = “GeoTweet”;
string sLog = “Application”;

if (!EventLog.SourceExists(sSource))
EventLog.CreateEventSource(sSource, sLog);

EventLog.WriteEntry(sSource, ex.ToString());
}
}
}

Then, of course, I had to change the Ajax handler to simply read the value of FireHose.Tweet

while (string.IsNullOrEmpty(FireHose.Tweet))
{
Thread.Sleep(100);
}
Response.Write(FireHose.Tweet);
FireHose.Tweet = “”;

The Thread.Sleep allows the Ajax handler wait until a new tweet has arrived before writing to the output stream. Do note, that if there were more than one client visiting the page at the same time, then it would be a first-come-gets-the-tweet. Which is not ideal for most situations, but ok for my test.

**Update**

Working on a solution for multi-client access, so that one client would not “steal” the tweets that should be visible to all clients, I decided to add an extra field in the Ajax called “IgnoreTweetIf”, this is set to the length of the tweet (the full data, not the text only), and the Ajax handler will not return a tweet if the length of the last tweet it has matches the length of the tweet that the client last received. On first call this “IgnoreTweetIf” is set to zero, so that any tweet is returned.

Therefore, my Ajax handler is modified as follows;

var strIgnoreTweetIf = Request.QueryString[“IgnoreTweetIf”];
var intIgnoreTweetIf = Convert.ToInt16(strIgnoreTweetIf);
while (string.IsNullOrEmpty(FireHose.Tweet) || intIgnoreTweetIf == FireHose.Tweet.Length)
{
Thread.Sleep(100);
}
Response.Write(FireHose.Tweet);

And the Ajax code is modified thus;

function ajaxLoop(ignoreTweetIf) {
$.get(“NextTweet.aspx?IgnoreTweetIf=” + ignoreTweetIf, function (data) {
handleTweet(data);
ajaxLoop(data.length);
});
}

 

Advertisements
Categories: Uncategorized
  1. April 10, 2013 at 1:21 pm

    As an addition to this post; if you want to deserialize the JSON from Twitter into a useable object on the server-side, then you can use this object: (Call Tweet.Deserialize)

    using System.Collections.Generic;
    using System.Web.Script.Serialization;

    public class User
    {
    public int? id { get; set; }
    public string id_str { get; set; }
    public string name { get; set; }
    public string screen_name { get; set; }
    public string location { get; set; }
    public object url { get; set; }
    public string description { get; set; }
    public bool? @protected { get; set; }
    public int? followers_count { get; set; }
    public int? friends_count { get; set; }
    public int? listed_count { get; set; }
    public string created_at { get; set; }
    public int? favourites_count { get; set; }
    public object utc_offset { get; set; }
    public object time_zone { get; set; }
    public bool? geo_enabled { get; set; }
    public bool? verified { get; set; }
    public int? statuses_count { get; set; }
    public string lang { get; set; }
    public bool? contributors_enabled { get; set; }
    public bool? is_translator { get; set; }
    public string profile_background_color { get; set; }
    public string profile_background_image_url { get; set; }
    public string profile_background_image_url_https { get; set; }
    public bool? profile_background_tile { get; set; }
    public string profile_image_url { get; set; }
    public string profile_image_url_https { get; set; }
    public string profile_banner_url { get; set; }
    public string profile_link_color { get; set; }
    public string profile_sidebar_border_color { get; set; }
    public string profile_sidebar_fill_color { get; set; }
    public string profile_text_color { get; set; }
    public bool? profile_use_background_image { get; set; }
    public bool? default_profile { get; set; }
    public bool? default_profile_image { get; set; }
    public object following { get; set; }
    public object follow_request_sent { get; set; }
    public object notifications { get; set; }
    }

    public class Geo
    {
    public string type { get; set; }
    public List coordinates { get; set; }
    }

    public class Coordinates
    {
    public string type { get; set; }
    public List coordinates { get; set; }
    }

    public class BoundingBox
    {
    public string type { get; set; }
    public List<List<List>> coordinates { get; set; }
    }

    public class Attributes
    {
    }

    public class Place
    {
    public string id { get; set; }
    public string url { get; set; }
    public string place_type { get; set; }
    public string name { get; set; }
    public string full_name { get; set; }
    public string country_code { get; set; }
    public string country { get; set; }
    public List polylines { get; set; }
    public BoundingBox bounding_box { get; set; }
    public Attributes attributes { get; set; }
    }

    public class UserMention
    {
    public string screen_name { get; set; }
    public string name { get; set; }
    public int? id { get; set; }
    public string id_str { get; set; }
    public List indices { get; set; }
    }

    public class Entities
    {
    public List hashtags { get; set; }
    public List urls { get; set; }
    public List user_mentions { get; set; }
    }

    public class Tweet
    {
    public string created_at { get; set; }
    public long? id { get; set; }
    public string id_str { get; set; }
    public string text { get; set; }
    public string source { get; set; }
    public bool? truncated { get; set; }
    public long? in_reply_to_status_id { get; set; }
    public string in_reply_to_status_id_str { get; set; }
    public int? in_reply_to_user_id { get; set; }
    public string in_reply_to_user_id_str { get; set; }
    public string in_reply_to_screen_name { get; set; }
    public User user { get; set; }
    public Geo geo { get; set; }
    public Coordinates coordinates { get; set; }
    public Place place { get; set; }
    public object contributors { get; set; }
    public int? retweet_count { get; set; }
    public int? favorite_count { get; set; }
    public Entities entities { get; set; }
    public bool? favorited { get; set; }
    public bool? retweeted { get; set; }
    public string filter_level { get; set; }
    public string lang { get; set; }

    ///

    /// Converts a JSON string into an object of type T.
    ///

    ///
    /// The JSON string.
    ///
    public static T Deserialize(string json)
    {
    json = json.Replace(“\”__type\””, “\”TypeName\””);
    var jsSerializer = new JavaScriptSerializer();
    return jsSerializer.Deserialize(json);
    }

    }

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: