Network Programming in .NET

To run Windows on a mac, you can run a virtual machine, or make it dual boot. However, those approaches use a load of disk space, and memory. There is another approach, which would be obvious to anyone from the Linux world, but it also applies to OSX. – It’s called Wine, and it is available for download here; https://wiki.winehq.org/MacOS

You will also need to download XQuartz, which is an X11 windows server, and is available here; https://www.xquartz.org/

With that, you can then simply right click your exe, and press Send to > Wine Stable, and it might just run. The interface looks quite different, and some applications I tried didn’t work.

If you are writing a Dialogflow webhook in node, you’re probably using dialogflow-fulfillment , and if you’re reading this post, you’ve run into the issue where DialogFlow is not returning properly when you carry out an asynchronous call.

Let’s see a typical syncronous case first;

intentMap.set(‘Default Welcome Intent’, welcome);

….

function welcome(agent) {
agent.add(“Welcome”);
}

Which returns Welcome, when the Default Welcome Intent is fired.

In this next typical use case, where I want to record failed queries, i.e. queries that trigger the fallback intent in a database. The Database is asyncronous, so it will fire a callback once the data is stored.

intentMap.set(‘Default Fallback Intent’, fallback);

….

function fallback(agent) {
database.recordFailure(request.body.queryResult.queryText, function(text)
{
agent.add(“Sorry, I couldn’t help you”);
});
}

Now, if you run this with the fallback intent, then you may notice that the data is stored in the database, but you get an error response back, rather than the “Sorry, I couldn’t help you” message

The trick is, to wrap the function in a promise like so;

function fallback(agent) {
return new Promise ( (resolve, reject) => {
database.recordFailure(request.body.queryResult.queryText, function(text)
{
agent.add(text);
return resolve();
});
});
}

And call resolve() once you’re ready to reply to the user.

If you have a lambda function above a certain size, (50 MB ish), then the inline code editor will no longer be available in AWS lambda. You then have to resort to uploading zipped files of your lambda function. This can be cumbersome on windows, if you haven’t got a quick script to run to zip and upload the lambda function.

You will need to have run AWS configure, and given this the correct credentials, and you will need to replace the function name below

@echo off
erase Lambda.zip
“C:\Program Files\7-Zip\7z” a Lambda.zip
aws lambda update-function-code –function-name xxxxxxx –zip-file “fileb://lambda.zip”

This ran in about 17 seconds for my lambda function, which would have taken easily a minute using the GUI. – plus, it’s less prone to human error.

If you hold a database of products or services priced in one currency, you might make it easier for your customers to display the offers in their own currency. But, you need to keep your exchange rates up to date, otherwise a sudden dip in a currency might mean that you loose money, or quote the wrong price to consumers.

There are plenty of free, and accurate feeds for currencies. I chose to use one from the ECB (European Central Bank), which I would trust.

Here is the XML Feed url:

https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml

Now, we want to pull that data on a daily basis, and import it into our database, using a serverless architecture on AWS. Using Lambda, Node, and I’m using SQL server as the database.

First off, in order to trigger a Lambda function on a schedule, you need to set up another service called cloudwatch. You create a cloudwatch rule, and set a schedule for one day – but for debugging purposes, set it to one minute initially, until the system is working. You connect this rule to your Lambda Function.

Now, I installed a number of NPM packages, Request, xml2Js, and MSsql. Those are to make HTTP requests easier, help handling XML, and connecting to SQL server, respectively.

Now, to set up my database, I created a table:

Create Table ExchangeRates
(
Currency char(3),
Rate float,
DateUpdated datetime default getdate()
)

The DateUpdated, is not strictly necessary, but it helps me see that the update function is running reliably.

I have to write some node code to execute my SQL statement, which I place in a file called database.js

const sql = require(‘mssql’);

(function() {

// ——- Database support ——- //

module.exports.execute = function (sqlQuery, callback)
{
const config = {
user: ‘xxxxx’,
password: ‘xxxx’,
server: ‘xxx.xxx.xxx’,
database: ‘xxxxx’,
};
sql.connect(config, (err) => {
if (err) {
console.log(err);
callback(err);
} else {
const req = new sql.Request();
req.query(sqlQuery, (error, result) => {
if (error) {
console.log(error);
callback(error);
} else {
console.log(result);
sql.close();
callback(null, result.recordset);
}
});
}
});

sql.on(‘error’, (err) => {
console.log(err);
callback(err);
});

};

}());

Now, I can simply say database.execute( SQL , callback) and the SQL will run.

Now, to make the request, parse the XML, and convert it into a SQL statement, I use this code;

function GetRatesFromECB(callback)
{
var strUrl = “https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml”;
request(strUrl, function (error, response, body) {
console.log(‘error:’, error);
console.log(‘statusCode:’, response && response.statusCode);
xml2js.parseString(body, function (_err, result) {
var strSQL = “truncate table ExchangeRates; \r\n”;
var cube = result[“gesmes:Envelope”][“Cube”][0][“Cube”][0][“Cube”];
for(var i in cube)
{
var exchange = cube[i][“$”];
strSQL += “insert into ExchangeRates (currency,rate) values (‘” + exchange.currency + “‘,” + exchange.rate + “); \r\n”;
console.log(exchange);
}
database.execute(strSQL, function(){
callback();
});
});
});
}

What this does, is that it reads all the rates from the XML file, then creates an SQL statement that wipes the table, and re-populates it with the latest rates.

Then, the standard boilerplate entry point is as follows

exports.handler = (event, context, callback) => {
GetRatesFromECB(function(){
callback(null, “OK”);
});
};

One of the key elements of serverless computing, is that you probably end up making requests to a third party domain via ajax to do any dynamic processing. In this case I’m imagining you are calling a Lambda function on AWS from a static website hosted on S3.

But, unless you enable CORS (Cross origin resource sharing) then the browser will prevent this type of AJAX calls being made.

So, you will need to do two things to enable CORS on AWS. The first, it log into API gateway, and select Enable CORS on the Actions drop down.

However, this is only half of the battle, since it enables CORS on the OPTIONS preflight request, you need to also set the CORS header in your lamda function, for which I use the following helper function;

function cors(data)
{
var response = {
statusCode: 200,
headers: {
‘Access-Control-Allow-Origin’: ‘*’,
‘Access-Control-Allow-Credentials’: true,
},
body: JSON.stringify(data),
};
return response;
}

Then, whenever you are returning from your lambda, you write something akin to;

callback(null, cors(“Hello World”));

In order to connect to Microsoft SQL server from Node in AWS Lambda you need to include a reference to require(‘mssql’); – but how do you install NPM packages on AWS Lambda.

That’s where the AWS CLI comes in. Download it, and the first thing you do is configure it with an IAM user as follows

aws configure

which prompts;

AWS Access Key ID [****************]:
AWS Secret Access Key [****************]:
Default region name [None]: eu-west-1
Default output format [None]:

Now, get a list of lambda functions as follows;

aws lambda list-functions

You can get the function name from the json returned, then if you want to get a zip of your function call;

aws lambda get-function –function-name <Function Name>

The JSON will return a property “Location” which you use to download the zip file.  Unzip the zip file, and navigate into the folder, then type

npm install mssql

Once complete, zip up the folder, and upload it back to AWS as follows

aws lambda update-function-code –function-name <Function Name> –zip-file “fileb://<ZIP Name>”

Now, you can create a file, database.js in your Lambda project as follows

const sql = require(‘mssql’);

(function() {

// ——- Database support ——- //

module.exports.execute = function(sqlQuery, callback)
{
const config = {
user: ‘xxxxxx’,
password: ‘xxxxx’,
server: ‘xxxx.xx.xx.xxx’,
database: ‘xxxxxx’,
};
sql.connect(config, (err) => {
if (err) {
console.log(err);
callback(err);
} else {
const req = new sql.Request();
req.query(sqlQuery, (error, result) => {
if (error) {
console.log(error);
callback(error);
} else {
console.log(result);
sql.close();
callback(null, result.recordset);
}
});
}
});

sql.on(‘error’, (err) => {
console.log(err);
callback(err);
});

};
}());

It is often surprising how busy your home network is, even when you think nothing much is happening on it, our homes are filled with smart devices that chat constantly on the network, constantly announcing technical details about themselves. If you run Wireshark on your computer, you can see just how noisy the network is.

This bit of code sniffs packets off the network, using a Arduino and a W5100 based ethernet shield. Note that this is  will only pick up packets directed at the arduino, or broadcast packets, and since the serial interface is way slower than your network connection, expect plenty of dropped packets.

I’ve got the output to show human-readable text where possible, – filtering out the null character (0), which interfered with copy and pasting the output. There’s not much intellegence going on here, so there’s going to be IPv4 mixed with IPv6 and ARP frames here.

The code is based on Nicolas Humfrey’s W5100 Mac Raw project here https://github.com/njh/W5100MacRaw

#include “w5100.h”

const byte mac_address[] = {
0xae, 0x03, 0xf3, 0xc7, 0x08, 0x78
};

Wiznet5100 w5100;

void setup() {
// Setup serial port for debugging
Serial.begin(115200);
w5100.begin(mac_address);
}
uint8_t buffer[800];
void loop() {
uint16_t len = w5100.readFrame(buffer, sizeof(buffer));
if ( len > 0 ) {
for(uint16_t i=0; i<len; i++)
{
if (buffer[i]>0)
{
Serial.print((char)buffer[i]);
}
}
Serial.println();
}
}

For more information on Packet Sniffing, check out chapter 10 in my book

http://webtropy.com/articles/book-network-programming-art6-10.asp

The UK has 30 Million cars, and this database covers 24.5 Million of them. Cars under 3 years old are not available, nor are Northen Irish cars.

Fields available are

• registration
• make
• model
• firstUsedDate
• fuelType
• primaryColour
• registrationDate
• manufactureDate
• engineSize
• LastMOT
• LastTestResult
• LastOdometerValue
• LastOdometerUnit

Anyone interested should contact via https://www.regcheck.org.uk/

In order to extend the capabilities of the Arduino, you add what are known as Shields. which are stackable boards that plug into the top of the Arduino board.

Ethernet support is one key feature missing in the Arduino board, so I bought a HiLetGo Ethernet shield from Amazon for £8. It’s based on the W5100 chip, and pretty standard kit.

You plug the shield into the top of the arduino, then connect the USB to the computer, and the ethernet cable to your router.

My project was to get the public IP address of my router – using the ipify API. The code is based on the TCP/IP code listed on https://www.arduino.cc/en/Tutorial/WebClient

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

// if you don’t want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(74,125,232,128); // numeric IP for Google (no DNS)
char server[] = “api.ipify.org”; // name address for ipify.com (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192, 168, 0, 177);
IPAddress myDns(192, 168, 0, 1);

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

// Variables to measure the speed
unsigned long beginMicros, endMicros;
unsigned long byteCount = 0;
bool printWebData = true; // set to false for better speed measurement

void setup() {
// You can use Ethernet.init(pin) to configure the CS pin
//Ethernet.init(10); // Most Arduino shields
//Ethernet.init(5); // MKR ETH shield
//Ethernet.init(0); // Teensy 2.0
//Ethernet.init(20); // Teensy++ 2.0
//Ethernet.init(15); // ESP8266 with Adafruit Featherwing Ethernet
//Ethernet.init(33); // ESP32 with Adafruit Featherwing Ethernet

// Open serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}

// start the Ethernet connection:
Serial.println(“Initialize Ethernet with DHCP:”);
if (Ethernet.begin(mac) == 0) {
Serial.println(“Failed to configure Ethernet using DHCP”);
// Check for Ethernet hardware present
if (Ethernet.hardwareStatus() == EthernetNoHardware) {
Serial.println(“Ethernet shield was not found. Sorry, can’t run without hardware. :(“);
while (true) {
delay(1); // do nothing, no point running without Ethernet hardware
}
}
if (Ethernet.linkStatus() == LinkOFF) {
Serial.println(“Ethernet cable is not connected.”);
}
// try to congifure using IP address instead of DHCP:
Ethernet.begin(mac, ip, myDns);
} else {
Serial.print(” DHCP assigned IP “);
Serial.println(Ethernet.localIP());
}
// give the Ethernet shield a second to initialize:
delay(1000);
Serial.print(“connecting to “);
Serial.print(server);
Serial.println(“…”);

// if you get a connection, report back via serial:
if (client.connect(server, 80)) {
Serial.print(“connected to “);
Serial.println(client.remoteIP());
// Make a HTTP request:
client.println(“GET /?format=json HTTP/1.1”);
client.println(“Host: api.ipify.org”);
client.println(“Connection: close”);
client.println();
} else {
// if you didn’t get a connection to the server:
Serial.println(“connection failed”);
}
beginMicros = micros();
}

void loop() {
// if there are incoming bytes available
// from the server, read them and print them:
int len = client.available();
if (len > 0) {
byte buffer[80];
if (len > 80) len = 80;
client.read(buffer, len);
if (printWebData) {
Serial.write(buffer, len); // show in the serial monitor (slows some boards)
}
byteCount = byteCount + len;
}

// if the server’s disconnected, stop the client:
if (!client.connected()) {
endMicros = micros();
Serial.println();
Serial.println(“disconnecting.”);
client.stop();
Serial.print(“Received “);
Serial.print(byteCount);
Serial.print(” bytes in “);
float seconds = (float)(endMicros – beginMicros) / 1000000.0;
Serial.print(seconds, 4);
float rate = (float)byteCount / seconds / 1000.0;
Serial.print(“, rate = “);
Serial.print(rate);
Serial.print(” kbytes/second”);
Serial.println();

// do nothing forevermore:
while (true) {
delay(1);
}
}
}

Note, that the Mac address listed in the code is really just a random number, as long as it’s unique on your network, it doesn’t cause a problem… and it’s extremely likely to be unique.

The output appears in the serial window such as the following;

This is a really beginers guide to the most simple thing that you can possibly do with an Arduino – Say Hello World.

So, first thing is, go to https://www.arduino.cc/en/Main/Software and download the Windows IDE. This installs all the drivers, and gives you a nice coding interface.

Once loaded, connect the Arduino to your PC using a USB cable. You don’t need a power cable, the USB will provide the power.

Now, edit the code to say the following

void setup() {
// put your setup code here, to run once:
Serial.begin(9600); // open the serial port at 9600 bps:
}

void loop() {
// put your main code here, to run repeatedly:
Serial.print(“Hello World\n”); // prints a label
}

Now, press Verify and Upload.

Once Uploaded, then press Tools > Serial Monitor, and you should see a long line of “Hello World”‘s