Archive

Author Archive

Create your own Flash Briefing with #Alexa #Skills

So, this started because I wanted to listen to Italian news (RAI) on an English (UK) Amazon Alexa device, and the news source was not available. With a quick Google, I found someone who released a NodeJS based app that captured the feed from RAI, and reformatted the JSON into a format that is compatible with Alexa. I forked this repo here;

https://github.com/infiniteloopltd/alexa-flash-briefing-grr-radio-rai

Then, using Heroku, I deployed the Github repo onto a temporary domain, which you can see here;

https://flash-briefing.herokuapp.com/

Then, I headed over to Developer.Amazon.com, clicked on Alexa -> Create Alexa Skills -> Console

Then Create Skill -> Enter a Name (Italy News) – > Select a Language (English (UK)) ;

This should match your Alexa device exactly. English US and English UK are different !

Select Flash Briefing -> Create Skill

Add an error message , like “Sorry, failed”

Press “Add new Feed”

Fill in the fields, like preamble, name, Update frequency. The Content type should be audio, and the feed URL should be the Heroku Url above.

Then, from your alexa app in your phone, click More -> Settings -> Flash Briefing, and your new source should be in the list

Now, you’ll have a new news source when you say “Play News” to Alexa

Categories: Uncategorized

Storing temporary data in #Redis in C#

Redis is an in-memory database, designed for ultra-fast data retrieval. It’s great for short-lived data, that perhaps you only need to store for the duration of a user session, or transaction.

The AWS version, of Redis, under the name “ElasticCache” can only be accessed from within the AWS network, or specifically the Redis VPC. This is obviously designed to enforce recommended usage. You get no performance advantage if your Web server needs to traverse the Internet to reach your Redis server.

Here, I’ve used RedisCloud Hosted Redis database, rather than a local installation. But it has the advantage that it can be accessed from anywhere. Good for development, not designed for production. The key is in plain text, feel free to mess about with the server.

So, this was my use case: I wanted to store data temporarily, just for 2 seconds, and then delete it. It’s actually rather non-trivial with a standard MySQL database to do this in a scaleable way.

So, step 1 is to import a client library, here I picked StackExchange.Redis;

Install-Package StackExchange.Redis

Now, I connect to the Redis server, and write a value that will expire in 2 seconds;

const string endpoint = "redis-15557.c72.eu-west-1-2.ec2.cloud.redislabs.com:15557,password=JU455eaOlQZjVYExorUl1oFouO509Ptu";
var redis = ConnectionMultiplexer.Connect(endpoint);
var db = redis.GetDatabase();

const string setValue = "abcdefg";
db.StringSet("mykey", setValue, TimeSpan.FromSeconds(2));
    

If I read the value back instantly, I get the expected value of “abcdefg”. If I wait 3 seconds and try to read again, I get null;

string getValue = db.StringGet("mykey");
Console.WriteLine(getValue); // writes: "abcdefg"
Thread.Sleep(TimeSpan.FromSeconds(3));
string getValue2 = db.StringGet("mykey");
Console.WriteLine(getValue2); // writes nothing

The code is available to clone here; https://github.com/infiniteloopltd/RedisTest

Categories: Uncategorized

Run #postman collections in C# (.NET Core) hosted on AWS Lambda

Major Caveat, this is very much a work in progress, that I hope to complete one day, or that someone will complete, and be nice enough to share the code via a pull request.

But, here’s the repo on github: https://github.com/infiniteloopltd/PostmanSharp

The motivation behind this, was a way to define a Javascript library that could be used to execute Postman collections. Javascript can’t call most APIs, apart from those with CORS enabled, or hosted on the same server. However, Postman offers a nice way to define an API, and export that definition as a Postman collection.

So, what I did, was created a C# library that interprets a Postman collection, and carries out the calling of that API, using variables passed in. So far, It only does HTTP GET requests, but it could be expanded easily.

So, let’s start off with the library; which is defined as follows –

public class Postman
{
	private JObject postmanCollection;
	public Postman(string collection)
	{
	    postmanCollection = JObject.Parse(collection);
	}

	public string Execute(string function, string variables)
	{
	    var item = postmanCollection["item"].FirstOrDefault(
		j => j["name"] + "" == function);
	    if (item == null) throw new ArgumentException("function not recognized");
	    var request = item["request"];
	    var method = request["method"] + "";
	    if (method != "GET") 
		throw new NotImplementedException("Only HTTP GET is currently supported");
	    if (request["header"] is JArray headers && headers.Count > 0) 
		throw new NotImplementedException("HTTP Headers are not yet supported");
	    var url = request["url"]["raw"] + "";
	    var jVariables = JObject.Parse(variables);
	    foreach (var (key,value) in jVariables)
	    {
		url = url.Replace("{{" + key + "}}", value+"");
	    }
	    using var web = new WebClient {Encoding = Encoding.UTF8};
	    var result = web.DownloadString(url);
	    return result;
	}
}

You could use this from a console app, to call a postman-defined API, but to make it more interesting, I created an AWS Lambda function, and configured my API gateway to permit CORS with the following code:

public string FunctionHandler(LambdaRequest request, ILambdaContext context)
{
    string result;
    try
    {
	LambdaLogger.Log("FunctionHandler called");
	LambdaLogger.Log(request.body);
	if (!request.body.StartsWith("{"))
	{
	    request.body = Encoding.UTF8.GetString(Convert.FromBase64String(request.body));
	}
	var jsonBody = JObject.Parse(request.body);
	var collection = jsonBody["collection"] + "";
	var function = jsonBody["function"] + "";
	var variables = jsonBody["variables"] + "";
	var postman = new Postman(collection);
	result = postman.Execute(function, variables);
    }
    catch (Exception ex)
    {
	result = ex.ToString();
    }
    LambdaLogger.Log(result);
    return result; 
}

Where LambdaRequest is defined simply as:

public class LambdaRequest
{
    public string body { get; set; }
}

Now, once the Lambda is uploaded to AWS, and an API gateway, with CORS enabled is set up, then I defined my Javascript class as follows (actual url omitted)

class Postman
{
	constructor(collection)
	{
		this.collection = collection;
	}

	execute(postmanFunction, variables)
	{
		var url = "https://xxxxx.amazonaws.com/default/Postman";
		return this.postData(url, 
			{
			 "collection" : this.collection, 
			 "function" : postmanFunction, 
			 "variables" : JSON.stringify(variables)
			});
	}

	async postData(url = '', data = {}) {
	  const response = await fetch(url, {
		method: 'POST', 
		mode: 'cors', 
		body: JSON.stringify(data) 
	  });
	  return response.json(); 
	}

}

Then, it’s called something like this in the page;

var collection = ... 
var postman = new Postman(collection);
var variables = {
	address : "Dublin, Ireland"
};
postman.execute("Geolocation",variables).then( data => {
	console.log(data);
	var pos = data.Response.View[0].Result[0].Location.NavigationPosition[0];
	alert(pos.Latitude + "," + pos.Longitude);
});

Categories: Uncategorized

Record #Mp4 #H264 video from a webcam in C# (.NET Core)

This absolute masterpiece of a video was created with OpenCV and FFMediaToolkit in C#, to be honest, and you too can create a video like this using the Code posted on the Github repo here –

https://github.com/infiniteloopltd/WebcamDemo

(Fluffy Pink Flamingo not included)

This has thought me quite a bit about the underlying workings of the Bitmap file format, and I’m sure there is a better way to do this, I do welcome comments and suggestions below, but this may be helpful to someone.

Ok, first the basics –

My machine had two webcams, so I wanted to choose between them; therefore I used the DirectShowLib NuGet Package (Install-Package DirectShowLib) as follows –

private static int SelectCameraIndex()
{
	var cameras = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice);
	if (cameras.Length == 1) return 0;
	foreach (var (camera, index) in WithIndex(cameras))
	{
		Console.WriteLine($"{index}:{camera.Name}");
	}
	Console.WriteLine("Select a camera from the list above:");
	var camIndex = Convert.ToInt32(Console.ReadLine());
	return camIndex;
}

The “WithIndex” is a helper function that gives an indexer in a foreach loop, not essential, but elegant;

private static IEnumerable<(T item, int index)> WithIndex<T>(IEnumerable<T> source)
{
	return source.Select((item, index) => (item, index));
}

Now, what I wanted to do is initialize the capture device (the webcam) to capture images, and feed them into a Mp4 video for 5 seconds, then stop.

var camIndex = SelectCameraIndex();
_captureDevice = new VideoCapture(camIndex, VideoCapture.API.DShow)
	{FlipVertical = true};
_captureDevice.ImageGrabbed += CaptureDeviceImageGrabbed;
var settings = new VideoEncoderSettings(width: 
	_captureDevice.Width
	, height: _captureDevice.Height
	, framerate: 15
	, codec: VideoCodec.H264)
{
	EncoderPreset = EncoderPreset.Fast,
	CRF = 17 // Constant Rate Factor
};
// Download from https://github.com/BtbN/FFmpeg-Builds/releases
FFmpegLoader.FFmpegPath =
	@"C:\Users\fiach\source\repos\Webcam\FFmpeg\";
_videoOutput = MediaBuilder.CreateContainer(@"c:\temp\example.mp4").WithVideo(settings).Create();
_captureDevice.Start();
Thread.Sleep(TimeSpan.FromSeconds(5));
_captureDevice.Stop();
_captureDevice.Dispose();
_videoOutput.Dispose();

The FlipVertical setting, I will explain later, but effectively, I am telling the capture device (webcam) to trigger the CaptureImageGrabbed event every time a new image is available.

I am initialising a “Container” which will store it’s output at “C:\temp\example.mp4”. This container will be fed with images from the CaptureImageGrabbed event. The main thread sleeps for 5 seconds, as this capture-encode cycle happens, and once the thread wakes up, it stops the capture device, and cleans up the resources used.

So, lets look at the CaptureImageGrabbed event;

private static void CaptureDeviceImageGrabbed(object sender, System.EventArgs e)
{
	var frame = new Mat();
	_captureDevice.Retrieve(frame);
	var buffer = new VectorOfByte();
	var input = frame.ToImage<Bgr, byte>();
	CvInvoke.Imencode(".bmp", input, buffer);
	var bitmapData = buffer.ToArray();
	
	var headerLessData = RedBlueSwap(bitmapData.Skip(54).ToArray());
	var imageData = ImageData.FromArray(headerLessData, ImagePixelFormat.Rgb24, frame.Size);
	_videoOutput.Video.AddFrame(imageData);
}

This is where we deal with trying to mesh two incompatible image formats. The image captured from the camera, and the image required by the FFMPeg container. There are bound to be better ways to do this, but this is how I did this.

I retrieve the frame data from the camera, and then convert this into a Bitmap file format in memory, and store this in a buffer byte array. I then have to do some rather weird operations to convert the Bitmap file format into a image array used by FFMPeg.

The Bitmap image format has a 54 byte header, which can be removed by calling the Skip(54) method. If you don’t do this step, you get an error saying “‘Pixel buffer size doesn’t match size required by this image format.'”

The Bitmap image format also is written backwards, which means the image is upside down – hence the “FlipVertical” used in the capture device. Since it is backwards, it also means that the Red and Blue colour channels are reversed. This does create an interesting colour effect, and took me a while to work out what was wrong!

This is the code for the Red – Blue Swap;

private static byte[] RedBlueSwap(byte[] input)
{
	var output = new byte[input.Length];
	for (var i = 0; i < input.Length - 3; i += 3)
	{
		var r = input[i];
		var g = input[i + 1];
		var b = input[i + 2];
		output[i] = b;
		output[i + 1] = g;
		output[i + 2] = r;
	}
	return output;
}

And that’s all there was to it. If you run this code, it will take a 5 second video, and store it locally.

You also have access to the image data as the video is being made, so you can adapt this to do motion detection – image recognition, real-time video editing, whatever you need!

This version is for Windows, but the components used do have Linux versions also, so this will be a future project.

Categories: Uncategorized

Capture a #webcam image using .NET Core and #OpenCV

TL;DR; Here is the public github repo: https://github.com/infiniteloopltd/WebcamDemo

Using OpenCV to capture a webcam in .NET is an easy process, you just need to include two Nuget Packages;

Install-Package Emgu.CV 
Install-Package Emgu.CV.runtime.windows

The windows runtime is required if running this code on Windows, you will need to target another platform, then you could use “Emgu.CV.runtime.ubuntu” for instance.

Here, you can use VideoCapture object to capture your video:

static void Main(string[] args)
{
  var filename = "webcam.jpg";
  if (args.Length > 0) filename = args[0];
  using var capture = new VideoCapture(0, VideoCapture.API.DShow); 
  var image = capture.QueryFrame(); //take a picture
  image.Save(filename);
}

The parameters to the VideoCapture constructor are the video source index (0 being default), and the API being used, here DirectShow worked best for me.

Categories: Uncategorized

Exploring #Whois data in C#

When you register a domain name, your details are kept in a Whois file with your registrar, which is then accessible via a Whois query.

What is interesting is that the format of the response of Whois is very unstandardized, and varies both in form and content from country to country. I checked this with a few of my own domains, to see the difference in the raw response;

Ireland:

https://github.com/infiniteloopltd/WhoisQuery/blob/master/ie-tld.txt

Mostly redacted, the organisation name was visible though.

Island

https://github.com/infiniteloopltd/WhoisQuery/blob/master/is-tld.txt

The details of the registrar were available, but just a handle for the registrant.

Italy

https://github.com/infiniteloopltd/WhoisQuery/blob/master/it-tld.txt

The registrant was visible, perhaps because it is a company, not a natural person.

Latvia

https://github.com/infiniteloopltd/WhoisQuery/blob/master/lv-tld.txt

No registrant detail available.

Ukraine

https://github.com/infiniteloopltd/WhoisQuery/blob/master/ua-tld.txt

The details of the registrar were available, but just a handle for the registrant. Text in Cyrillic (UTF8)

UK

https://github.com/infiniteloopltd/WhoisQuery/blob/master/uk-tld.txt

Minimal details of the registrar were available.

Israel

Omitted because of personal data availability

This was the only TLD that disclosed actual personal details, including a full name, and address.

The TLDs .com.cy (Cyprus) and .com.my (Malaysia) didn’t seem to work, when I tried.

Categories: Uncategorized

Car Registration #API available for #Malaysia

Malaysia has a car ownership percentage of 61%, and a population of 31 million, which is over 18.6 million vehicles. If your business intends to sell to this automotive market, one of the best tools to have at your disposal is an API lookup based on Malaysian Number plates. With this API, you can prompt your user to enter his or her number plate, and receive technical information about the vehicle that they own, to quickly assist your user with the product or service they are requesting.

Our API can be found here; http://www.vehicleapi.com.my/

Malaysia support


Car registration plates in Malaysia use the /CheckMalaysia  endpoint and return the following information:

  • Make / Model
  • Age
  • VIN number
  • Engine size
  • Representative image

Sample Registration Number: 

WXX7385

Sample Json:

{

  “Description”: “PERODUA VIVA ELITE EXCLUSIVE EJ VE 4 SP AUTOMATIC”,

  “RegistrationYear”: “2013”,

  “CarMake”: {

    “CurrentTextValue”: “PERODUA”

  },

  “CarModel”: {

    “CurrentTextValue”: “VIVA”

  },

  “MakeDescription”: {

    “CurrentTextValue”: “PERODUA”

  },

  “ModelDescription”: {

    “CurrentTextValue”: “VIVA”

  },

  “Seats”: “5”,

  “Body”: “HATCHBACK”,

  “Fuel”: “”,

  “Transmission”: “”,

  “VIN”: “PM2L251S002204297”,

  “NVIC”: “HQS13A”,

  “EngineNumber”: “L58B67A”,

  “EngineSize”: “989”,

  “Drive”: “”,

  “Insurance”: {

    “Insurer”: “RHB INSURANCE BERHAD”,

    “CoverType”: “Comprehensive”,

    “PolicyStatus”: “Active”,

    “PolicyNumber”: “D20MPCP3155505KT”

  },

  “ImageUrl”: “http://www.vehicleapi.com.my/image.aspx/@UEVST0RVQSBWSVZB&#8221;

}

Categories: Uncategorized

Implementing #EV #X509 parsing in C# (.NET Core)

Security certificates can be DV (Domain Validation), OV (Organisation Validation) or EV (Enhanced Validation).

DV will prove that the domain you are visiting is what it says it is, but says nothing about the organisation that runs the domain. It’s perfectly possible for a hacker legally purchase the domain “Micr0s0ft.com” (note the zero), and get a DV SSL cert for it. The DV is not saying that the hacker has any relation to the company “Microsoft Inc”, it’s just saying that the domain has been externally validated to respond to a basic ownership challenge.

OV and EV go a step further, and the certificate issuer will take extra manual steps to verify, that the domain is owned by a given company, and/or at a particular address.

This level of EV validation is then stored in the Subject line of the Certificate, and can be read by all visitors of the site. Browsers will typically highlight the extra level of trust and verification with a green tick in the address bar.

Reading and parsing the subject line can be done with some C# code shown at this repo; https://github.com/infiniteloopltd/EvCertParser

The code to get the cert is as follows;

public async Task<EvCertificate> Request(string url)
{
	EvCertificate certificate = null;
	var handler = new HttpClientHandler
	{
		UseDefaultCredentials = true,
		ServerCertificateCustomValidationCallback = (sender, cert, chain, error) =>
		{
			var export = cert.Export(X509ContentType.SerializedCert);
			certificate = new EvCertificate(export);
			return error == SslPolicyErrors.None;
		}
	};
	using var client = new HttpClient(handler);
	using var response = await client.GetAsync(url);
	return certificate;
}

Which can evidently throw errors if the URL is invalid, doesn’t have a cert, or an invalid cert.

Categories: Uncategorized

Making a DataReader behave like a DataSet #sql #dotNET

In C# / .NET a DataSet is easy to use, you can iterate over it using a simple For Loop, however, if you are using a large query like “select * from hugeTable” then you’ll find that the memory footprint, plus the time taken to read the whole dataset into memory is prohibitive.

Using a DataReader is the alternative, where you explicitly specify when you want the next row, so hopefully, the first row of your query will return faster. But, client code can’t use a simple for loop to iterate over the data.

So, here is my hybrid solution;

public static IEnumerable<Dictionary<string, object>> PopulateReader(string command)
{
	var sqlConnection = "---READ DSN FROM CONFIG---";
	var connection = new SqlConnection(sqlConnection);
	connection.Open();
	var sqlCommand = new SqlCommand(command, connection);
	var reader = sqlCommand.ExecuteReader();
	while (reader.Read())
	{
	    var result = new Dictionary<string, object>();
	    for (var column = 0; column < reader.FieldCount; column++)
		result.Add(reader.GetName(column), reader.GetValue(column));
	    yield return result;
	}
	connection.Close();
}

And your client can call this, in a very similar way to how a DataReader would be called.

var reader = PopulateReader(sql);
foreach (var dr in reader)
{
	var id = dr["id"].ToString();
	... 
}

Hope this helps, or acts as inspiration for someone!

Categories: Uncategorized

Retrieving state in a FirstChanceException in .NET

If you’re using a application-wide FirstChanceException handler, this is great for covering lots of bases at once, it helps make sure that you can catch (and record) an exception even if it happens with a bit of code you didn’t expect.

However, the scope of the handler is what it is, so accessing state, and hence, the context of the error, is going to be tricky. After all, it’s helpful to know where the error happened, but what was the state of your application at the time, is there an edge-case that causes the error?

So, Lets try and demonstrate this issue, in a simplistic example.

static void Main(string[] args)
{
    AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
    {
	Console.WriteLine("An Error occurred, but at what state?");
    };
    Parallel.For(-100, 100, (state) =>
	{
	    if (state == 42)
	    {   
		throw new ArgumentException("Some error with state #42");
	    }
	}
    );
    Thread.Sleep(TimeSpan.FromSeconds(5));
}

So, in this example, we can see that state number 42 causes an error. The FirstChance exception is triggered, but within the handler, we cannot tell what state causes the error.

If we try a na├»ve approach, and just try storing the state in a static variable, and then report the last known value, let’s see what happens;

private static int lastKnownState = 0;
static void Main(string[] args)
{
    AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
    {
	Console.WriteLine($"Last Known state {lastKnownState}");
	Console.WriteLine("An Error occurred, but at what state?");
    };
    Parallel.For(-100, 100, (state) =>
	{
	    lastKnownState = state;
	    if (state == 42)
	    {   
		throw new ArgumentException("Some error with state #42");
	    }
	}
    );
    Thread.Sleep(TimeSpan.FromSeconds(5));
}

In this case, “lastKnownState” returned as 99, not 42. It may be a different value on your computer, but it’s important to note that it’s not correct. Since the lastKnownState gets overwritten by multiple threads, and so it’s not thread-safe.

So Attempt #2, let’s create a concurrent Dictionary, store the states associated with each thread, then lookup the state within the FirstChanceExceptionHandler.

public static ConcurrentDictionary<int,int> ThreadStates = new ConcurrentDictionary<int, int>();
static void Main(string[] args)
{
    AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
    {
	var ok = ThreadStates.TryGetValue(Thread.CurrentThread.ManagedThreadId,out var state);
	if (ok) Console.WriteLine($"An error occurred at state {state}");
    };
    Parallel.For(-100, 100, state =>
	{
	    ThreadStates.TryAdd(Thread.CurrentThread.ManagedThreadId, state);
	    if (state == 42)
	    {	      
		throw new ArgumentException("Some error with state #42");
	    }  
	}
    );
    Thread.Sleep(TimeSpan.FromSeconds(5));
}

This looks right, but, it’s not. The state is still incorrect within the Exception Handler? Why is this. Well, because Parallel.For will recycle threads. Which means that the “ManagedThreadId” can be the same for multiple states. The TryAdd will ignore duplicates, so data will be lost.

public static ConcurrentDictionary<int,int> ThreadStates = new ConcurrentDictionary<int, int>();
static void Main(string[] args)
{
    AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
    {
	var ok = ThreadStates.TryGetValue(Thread.CurrentThread.ManagedThreadId,out var state);
	if (ok) Console.WriteLine($"An error occurred at state {state}");
    };
    Parallel.For(-100, 100, state =>
	{
	    ThreadStates.TryAdd(Thread.CurrentThread.ManagedThreadId, state);
	    if (state == 42)
	    {
	      
		throw new ArgumentException("Some error with state #42");
	    }
	    ThreadStates.TryRemove(Thread.CurrentThread.ManagedThreadId, out var oldState);

	}
    );
    Thread.Sleep(TimeSpan.FromSeconds(5));
}

So, the trick is to remove the state from the ThreadStates collection once complete, using the code highlighted in bold.

Now, that’s working, let’s refactor the code to keep the specifics of the code separate from the main logic flow. So, I’m creating a class called ThreadContext as follows;

public sealed class ThreadContext<T> : IDisposable
{
	private static readonly ConcurrentDictionary<int, T> ThreadStates = new ConcurrentDictionary<int, T>();

	public static T RetrieveState()
	{
	    var ok = ThreadStates.TryGetValue(Thread.CurrentThread.ManagedThreadId, out var state);
	    return ok ? state : default;
	}

	public void Dispose()
	{
	    ThreadStates.TryRemove(Thread.CurrentThread.ManagedThreadId, out var oldState);
	}

	public ThreadContext(T state)
	{
	    ThreadStates.TryAdd(Thread.CurrentThread.ManagedThreadId, state);
	}
}

Notice, that the class is now generic, so that it can hold any nullable object – like a string, a nullable int, or something much more complex.

Now the Main method becomes

AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
{
	var state = ThreadContext<int?>.RetrieveState();

	if (state != null) Console.WriteLine($"An error occurred at state {state}");
};
Parallel.For(-100, 100, state =>
{
    using (new ThreadContext<int?>(state))
    {

	if (state == 42)
	{
	    throw new ArgumentException("Some error with state #42");
	}
    }

}
);
Thread.Sleep(TimeSpan.FromSeconds(5));

Now to go try this on a real project …

Categories: Uncategorized