Network Programming in .NET

The major benifit of CouchDb, over relational databases, is that it forces you to use Map/Reduce, which  offers “out-of-the-box”, an easy path to scalability, in terms of raw data storage, performance and redundancy.

However, 10 CouchDB instances may be quicker than 10 SQL server boxes, but certainly, one instance is terribly slow on ~100K records at ~1 GB in size (total). In fact, it’s unusably slow 😦

Here’s an example I put live (http://couchdb.webtropy.com/), which returns 100 records at random from Delicious data – which is stored on a couchdb at couchone.com. However, the query was too slow to run a Group-by clause, nor could it even do a select, on so many records.

I modified SharpCouch to allow for limiting, grouping and skip functionality, as follows:

/// <summary>
/// Execute a temporary view and return the results.
/// </summary>
/// <param name=”server”>The server URL</param>
/// <param name=”db”>The database name</param>
/// <param name=”map”>The javascript map function</param>
/// <param name=”reduce”>The javascript reduce function or
/// null if not required</param>
/// <param name=”startkey”>The startkey or null not to use</param>
/// <param name=”endkey”>The endkey or null not to use</param>
/// <returns>The result (JSON format)</returns>
public string ExecTempView(string server,string db,string map,string reduce,string startkey,string endkey, bool Group, int Limit,int Skip)
{
// Generate the JSON view definition from the supplied
// map and optional reduce functions…
string viewdef=”{ \”map\”:\””+map+”\””;
if(reduce!=null)
viewdef+=”,\”reduce\”:\””+reduce+”\””;
viewdef+=”}”;

string url=server+”/”+db+”/_temp_view”;
if(startkey!=null)
{
url+=”?startkey=”+HttpUtility.UrlEncode(startkey);
}
if(endkey!=null)
{
if(startkey==null) url+=”?”; else url+=”&”;
url+=”endkey=”+HttpUtility.UrlEncode(endkey);
}
if (Group)
{
url += “?group=true”;
}
if (Limit > 0)
{
if (Group) url += “&”; else url += “?”;
url += “limit=” + Limit;
}
if (Skip > 0)
{
if (Group || Limit>0) url += “&”; else url += “?”;
url += “skip=” + Skip;
}

return DoRequest(url,”POST”,viewdef,”application/json”);
}

That at least gave me basic functionality.

I was hoping to do a group by author as follows:

function(doc) { emit(doc.author,1); }

function(keys, values, rereduce) {
return sum(values);
}

but that was too slow, and so too was a filter:

function(doc) {
if (doc.author == \”morgand\”)
emit(doc,1);
}

Overall, I was initially excited by it, and I was glad I could extend SharpCouch, but overall disapointed that it’s speed was awful.

Posing to this url:

https://docs.google.com/e/commonshare

app    2
authuser    0
clientUser    10495966446245
confirmed    false
foreignService    kix
hl    en
itemIds    1kKdL8JV8jJP5sDBprLzAgxDGvDTTA_s8KY
notificationInfo    {“messageType”:”invite”,”recipient”:”someemail@gmail.com“,”optionValues”:{“ccMe”:false,”pasteInEmail”:false}}
requestType    invite
role    30
sendEmail    true
shareProtocolVersion    2
shareService    kix
subapp    10
token    US5VKC0BAAA.J0BSgZ7BpREraHwLcA

&&&START&&&{“status”:0,”modelChanges”:[{“aclEntries”:[{“scope”:{“name”:”Their name appears here“,”id”:”114892889789266972″,”scopeType”:”user”,”me”:false,”email”:”somemail@gmail.com“},”role”:30}],”id”:”1kKdL8JV8jJP5Tr3prLzAgxDGvDTTA_s8KY”,”visibilitySetting”:{“role”:0,”summary”:”Private to me + 3 more”,”visibilityState”:”private”,”restrictedToDomain”:true,”visibilityEntries”:[{“role”:0,”summary”:”Private”,”visibilityState”:”private”,”restrictedToDomain”:true,”details”:”Only people explicitly granted permission can access. Sign-in required.”}]}}]}

Interesting? 🙂

I’ve decided to start moving some data from my MsSQL database to a MySQL one. Why?, because I’d like my MsSQL database to be used for my main websites, and the MySQL one used for less important ones. Those where the data is bulky but the performance isn’t so important. – Since my MySQL db is nowhere near my webserver.

Anyway, I was surprised how easy it was to migrate,

This is the code I use for MsSQL:

public DataSet ExecuteDataSet(string sql)
{
DataSet ds = new DataSet();
try
{

OleDbConnection DSN = new OleDbConnection(connectionString);
DSN.Open();
OleDbCommand Database = new OleDbCommand(sql,DSN);
OleDbDataAdapter Adapter = new OleDbDataAdapter(Database);
Adapter.Fill(ds,”sql”);
DSN.Close();
}
catch(Exception ex)
{
logError(ex,sql);
}
return ds;
}

Just replace OleDb with MySql and you get

public DataSet ExecuteMySqlDataSet(string sql)
{
DataSet ds = new DataSet();
try
{

MySqlConnection DSN = new MySqlConnection(MySqlConnectionString);
DSN.Open();
MySqlCommand Database = new MySqlCommand(sql, DSN);
MySqlDataAdapter Adapter = new MySqlDataAdapter(Database);
Adapter.Fill(ds, “sql”);
DSN.Close();
}
catch (Exception ex)
{
logError(ex, sql);
}
return ds;
}

And then include using MySql.Data.MySqlClient;

A few tweaks to the SQL, so instead of select top 10 * … it’s select * … limit 10, instead of order by newid() it’s order by rand(), and the table names are case sensitive.

I hit some wierd bug where it didn’t recognise column names with umlauts, but I just worked around that instead of fixing it. – Anyway, here’s the end-result: http://www.listofbanks.info/GermanBanks.aspx

I’ve made a page that locates your closest bank branch using a Geo-aware webpage see

http://www.listofbanks.info/FindBank.aspx

Once you share your location with the page, using the prompt provided, it makes an AJAX call with your longitude and latitude to another page which returns the top 10 banks in our database ordered by distance. The system will only work in europe, so americans may see their nearest bank in spain!.

The JavaScript I used was as follows:

if (geo_position_js.init()) {
geo_position_js.getCurrentPosition(success_callback, error_callback, { enableHighAccuracy: true });
}
else {
window.document.getElementById(“response”).innerHTML = “Failed.”;
}

function success_callback(p) {
var oXHR = new XMLHttpRequest();
var strUrl = “FindBankAjax.aspx?”;
strUrl += “Latitude=” + p.coords.latitude.toFixed(2);
strUrl += “&Longitude=” + p.coords.longitude.toFixed(2);
oXHR.open(“get”, strUrl, false);
oXHR.send(null);
window.document.getElementById(“response”).innerHTML = oXHR.responseText;
}

function error_callback(p) {
window.document.getElementById(“Failed due to ” + p.code)

}

Here’s a free MySQL database that anyone can use:

Database: wikienc1_free

Username: wikienc1_free

Password: 00036IT

Host: 184.154.116.162

FAQ:

1. Is this really free?

Yup. But I’d appreciate a backlink from your website!

2. Is it private?

Nope.

3. Won’t someone just delete my data

They could, but please don’t delete other people’s data.

4. Can’t you set up an account just for me?

Pay me and I will 🙂

5. Will you help install WordPress / Joomla / whatever ?

Nope.

6. Will the username or password change.

They could, bookmark this page, it will always be updated with the latest password

This is a nice little tool to ‘pretify’ C# code that’s displayed on a website, it’s all in Javascript, and doesn’t affect SEO, since the code is unchanged.

For example: http://reflector.webtropy.com/default.aspx/FXUpdate3074/FXUpdate3074/1@1/DEVDIV/depot/DevDiv/releases/whidbey/QFE/ndp/fx/src/xsp/System/Web/Compilation/PageBuildProvider@cs/1/PageBuildProvider@cs – Here I’ve used alex gorbatchev’s syntax highlighter, using the CSharp brush.

“Simples”

I’ve now put a simple interface on the source code viewer at http://reflector.webtropy.com, where you can type in the name of any .NET class, press search, and it shows the source code.

If you like it, please link to it, favourite it, or just buy my book for someone as a christmas present :).

I was interested to see that Microsoft have release portions of the source code for the .NET framework,see http://referencesource.microsoft.com/netframework.aspx

I’ve created an online viewer for this rather immense library,  see for example:

This is the source code to the DataContractJsonSerializer

http://reflector.webtropy.com/default.aspx/WCF/WCF/3~5~30729~1/untmp/Orcas/SP/ndp/cdf/src/NetFx35/System~ServiceModel~Web/System/Runtime/Serialization/Json/DataContractJsonSerializer~cs/3/DataContractJsonSerializer~cs

NUnit may say that a function works when run once, but what happens if you the same function twice at once. You may find that a static or shared dictionary could crash when adding two identical keys from two seperate threads.

So, here is a nice function to call another function multiple times at once, and verify the output.

/// <summary>
/// Puts a particular method under stress by calling it multiple times at once
/// </summary>
/// <typeparam name=”T”>The type of the parameter sent to the method under test</typeparam>
/// <typeparam name=”TResult”>The type of the result from the method.</typeparam>
/// <param name=”method”>The method.</param>
/// <param name=”parameter”>The parameter to be passed to the method.</param>
/// <param name=”check”>The check.</param>
/// <param name=”load”>The number of times the method should be called at once.</param>
public static void StressTest<T, TResult>(Func<T, TResult> method, T parameter, Predicate<TResult> check, int load)
{
var lWorkers = new List<IAsyncResult>();
for (var i = 0; i < load; i++)
{
lWorkers.Add(method.BeginInvoke(parameter, null, null));
}
foreach (var result in lWorkers.Select(method.EndInvoke))
{
Assert.IsTrue(check(result));
}
}

And that puts the funk into funktion 🙂

If you want to run Nunit testing on some ASP.NET code, then you might have some problems if the code being tested

makes reference to Request.Cookies. Here is how I got round it:

/// <summary>
/// Verify that critical parts of the default page are working
/// </summary>
[TestFixture]
public class DefaultMasterTesting
{
/// <summary>
/// Checks the Some Request.Cookies dependentent method
/// </summary>
[Test]
public void CheckSomeCookieDependentMethod()
{
var master = new Default();
master.Page = new Page();
var hrReq = new HttpRequest(“”, “http://localhost&#8221;, “”);
hrReq.Cookies.Add(new HttpCookie(“Cookie”, “Cookie Value”));
var hrResp = new HttpResponse(new StringWriter());
HttpContext.Current = new HttpContext(hrReq,hrResp);
var fiRequest = typeof (Page).GetField(“_request”, BindingFlags.NonPublic | BindingFlags.Instance);
fiRequest.SetValue(master.Page,HttpContext.Current.Request);
master.SomeCookieDependentMethod();
}
}

Interesting to see that Page.Request and HttpContext.Current.Request are different !