Archive

Archive for December, 2010

CouchDB: Extending SharpCouch with group, skip and limit

December 20, 2010 Leave a comment

The major benifit of CouchDb, over relational databases, is that it forces you to use Map/Reduce, which¬† offers “out-of-the-box”, an easy path to scalability, in terms of raw data storage, performance and redundancy.

However, 10 CouchDB instances may be quicker than 10 SQL server boxes, but certainly, one instance is terribly slow on ~100K records at ~1 GB in size (total). In fact, it’s unusably slow ūüė¶

Here’s an example I put live (http://couchdb.webtropy.com/), which returns 100 records at random from Delicious data – which is stored on a couchdb at couchone.com. However, the query was too slow to run a Group-by clause, nor could it even do a select, on so many records.

I modified SharpCouch to allow for limiting, grouping and skip functionality, as follows:

/// <summary>
/// Execute a temporary view and return the results.
/// </summary>
/// <param name=”server”>The server URL</param>
/// <param name=”db”>The database name</param>
/// <param name=”map”>The javascript map function</param>
/// <param name=”reduce”>The javascript reduce function or
/// null if not required</param>
/// <param name=”startkey”>The startkey or null not to use</param>
/// <param name=”endkey”>The endkey or null not to use</param>
/// <returns>The result (JSON format)</returns>
public string ExecTempView(string server,string db,string map,string reduce,string startkey,string endkey, bool Group, int Limit,int Skip)
{
// Generate the JSON view definition from the supplied
// map and optional reduce functions…
string viewdef=”{ \”map\”:\””+map+”\””;
if(reduce!=null)
viewdef+=”,\”reduce\”:\””+reduce+”\””;
viewdef+=”}”;

string url=server+”/”+db+”/_temp_view”;
if(startkey!=null)
{
url+=”?startkey=”+HttpUtility.UrlEncode(startkey);
}
if(endkey!=null)
{
if(startkey==null) url+=”?”; else url+=”&”;
url+=”endkey=”+HttpUtility.UrlEncode(endkey);
}
if (Group)
{
url += “?group=true”;
}
if (Limit > 0)
{
if (Group) url += “&”; else url += “?”;
url += “limit=” + Limit;
}
if (Skip > 0)
{
if (Group || Limit>0) url += “&”; else url += “?”;
url += “skip=” + Skip;
}

return DoRequest(url,”POST”,viewdef,”application/json”);
}

That at least gave me basic functionality.

I was hoping to do a group by author as follows:

function(doc) { emit(doc.author,1); }

function(keys, values, rereduce) {
return sum(values);
}

but that was too slow, and so too was a filter:

function(doc) {
if (doc.author == \”morgand\”)
emit(doc,1);
}

Overall, I was initially excited by it, and I was glad I could extend SharpCouch, but overall disapointed that it’s speed was awful.

 

 

Advertisements
Categories: Uncategorized

Google Security hole: Determine people’s names from their Gmail Account

December 20, 2010 Leave a comment

Posing to this url:

https://docs.google.com/e/commonshare

app    2
authuser    0
clientUser    10495966446245
confirmed    false
foreignService    kix
hl    en
itemIds    1kKdL8JV8jJP5sDBprLzAgxDGvDTTA_s8KY
notificationInfo¬†¬†¬† {“messageType”:”invite”,”recipient”:”someemail@gmail.com“,”optionValues”:{“ccMe”:false,”pasteInEmail”:false}}
requestType    invite
role    30
sendEmail    true
shareProtocolVersion    2
shareService    kix
subapp    10
token    US5VKC0BAAA.J0BSgZ7BpREraHwLcA

&&&START&&&{“status”:0,”modelChanges”:[{“aclEntries”:[{“scope”:{“name”:”Their name appears here“,”id”:”114892889789266972″,”scopeType”:”user”,”me”:false,”email”:”somemail@gmail.com“},”role”:30}],”id”:”1kKdL8JV8jJP5Tr3prLzAgxDGvDTTA_s8KY”,”visibilitySetting”:{“role”:0,”summary”:”Private to me + 3 more”,”visibilityState”:”private”,”restrictedToDomain”:true,”visibilityEntries”:[{“role”:0,”summary”:”Private”,”visibilityState”:”private”,”restrictedToDomain”:true,”details”:”Only people explicitly granted permission can access. Sign-in required.”}]}}]}

Interesting? ūüôā

 

Categories: Uncategorized

Moving from MsSQL to MySQL

December 20, 2010 Leave a comment

I’ve decided to start moving some data from my MsSQL database to a MySQL one. Why?, because I’d like my MsSQL database to be used for my main websites, and the MySQL one used for less important ones. Those where the data is bulky but the performance isn’t so important. – Since my MySQL db is nowhere near my webserver.

Anyway, I was surprised how easy it was to migrate,

This is the code I use for MsSQL:

public DataSet ExecuteDataSet(string sql)
{
DataSet ds = new DataSet();
try
{

OleDbConnection DSN = new OleDbConnection(connectionString);
DSN.Open();
OleDbCommand Database = new OleDbCommand(sql,DSN);
OleDbDataAdapter Adapter = new OleDbDataAdapter(Database);
Adapter.Fill(ds,”sql”);
DSN.Close();
}
catch(Exception ex)
{
logError(ex,sql);
}
return ds;
}

Just replace OleDb with MySql and you get

public DataSet ExecuteMySqlDataSet(string sql)
{
DataSet ds = new DataSet();
try
{

MySqlConnection DSN = new MySqlConnection(MySqlConnectionString);
DSN.Open();
MySqlCommand Database = new MySqlCommand(sql, DSN);
MySqlDataAdapter Adapter = new MySqlDataAdapter(Database);
Adapter.Fill(ds, “sql”);
DSN.Close();
}
catch (Exception ex)
{
logError(ex, sql);
}
return ds;
}

And then include using MySql.Data.MySqlClient;

A few tweaks to the SQL, so instead of select top 10 * … it’s select * … limit 10, instead of order by newid() it’s order by rand(), and the table names are case sensitive.

I hit some wierd bug where it didn’t recognise column names with umlauts, but I just worked around that instead of fixing it. – Anyway, here’s the end-result: http://www.listofbanks.info/GermanBanks.aspx

Categories: Uncategorized

How to make a Geo-Aware website?

December 17, 2010 Leave a comment

I’ve made a page that locates your closest bank branch using a Geo-aware webpage see

http://www.listofbanks.info/FindBank.aspx

Once you share your location with the page, using the prompt provided, it makes an AJAX call with your longitude and latitude to another page which returns the top 10 banks in our database ordered by distance. The system will only work in europe, so americans may see their nearest bank in spain!.

The JavaScript I used was as follows:

if (geo_position_js.init()) {
geo_position_js.getCurrentPosition(success_callback, error_callback, { enableHighAccuracy: true });
}
else {
window.document.getElementById(“response”).innerHTML = “Failed.”;
}

function success_callback(p) {
var oXHR = new XMLHttpRequest();
var strUrl = “FindBankAjax.aspx?”;
strUrl += “Latitude=” + p.coords.latitude.toFixed(2);
strUrl += “&Longitude=” + p.coords.longitude.toFixed(2);
oXHR.open(“get”, strUrl, false);
oXHR.send(null);
window.document.getElementById(“response”).innerHTML = oXHR.responseText;
}

function error_callback(p) {
window.document.getElementById(“Failed due to ” + p.code)

}

 

 

Categories: Uncategorized

Free MySQL database

December 15, 2010 Leave a comment

Here’s a free MySQL database that anyone can use:

 

Database: wikienc1_free

Username: wikienc1_free

Password: 00036IT

Host: 184.154.116.162

FAQ:

1. Is this really free?

Yup. But I’d appreciate a backlink from your website!

2. Is it private?

Nope.

3. Won’t someone just delete my data

They could, but please don’t delete other people’s data.

4. Can’t you set up an account just for me?

Pay me and I will ūüôā

5. Will you help install WordPress / Joomla / whatever ?

Nope.

6. Will the username or password change.

They could, bookmark this page, it will always be updated with the latest password

 

Categories: Uncategorized

Code syntax highlighter for C#

December 14, 2010 Leave a comment

This is a nice little tool to ‘pretify’ C# code that’s displayed on a website, it’s all in Javascript, and doesn’t affect SEO, since the code is unchanged.

For example: http://reflector.webtropy.com/default.aspx/FXUpdate3074/FXUpdate3074/1@1/DEVDIV/depot/DevDiv/releases/whidbey/QFE/ndp/fx/src/xsp/System/Web/Compilation/PageBuildProvider@cs/1/PageBuildProvider@cs – Here I’ve used alex gorbatchev’s syntax highlighter, using the CSharp brush.

“Simples”

Categories: Uncategorized

.NET Framework class source code viewer

December 11, 2010 Leave a comment

I’ve now put a simple interface on the source code viewer at http://reflector.webtropy.com, where you can type in the name of any .NET class, press search, and it shows the source code.

If you like it, please link to it, favourite it, or just buy my book for someone as a christmas present :).

Categories: Uncategorized