Intro - What is node.js


Node.js is a event based server side JS application. The reason that I refer to node as a application not a framework is because you write JavaScript not node. Node.js simply parses and runs your JavaScript while at the same time providing users with some core libraries with which to build applications with. In the background node uses the v8 JavaScript engine to extract the maximum performance out of JavaScript.

Server side JavaScript is nothing new , mozilla has been doing it for a while but it has never this accessible. Writing server side JavaScript means that you use JavaScript in a the way you would use any other programming language but for a function that it is really good at (dealing with events and callbacks). But the most important difference between normal browser based JavaScript and node is that you dont have a browser dom to work with , so familiar variables like window do not exist , instead you are given a new variables like process which gives you information about the current running node process. While you do no work with the browser it does not mean you can not work with web pages. There are a good few libraries that provide this functionality (more on this later).

Getting started with node.js is pretty easy if you are familiar with the unix way of building applications from source. If not this article should get you going pretty quickly, the basics are get the source, configure , build , build install. There are distribution for windows but it lags slightly behind the latest version , you could of course build it from source if you wanted to. Although you should keep in mind that unix is the preferred dev/working environment.

Once you have node up and running , you can check that everything is working by simply typing node at the command line. This will launch the node.js repl interpreter which is extremely useful for testing small chunks of code as you are working. To run a node script type node script.js at the console and node will execute your script.js file.

A simple console hello world can be seen below.

console example

One of the things that many people find hard to grasp is the non-blocking aspect of node. Non blocking is simply the concept of fire and forget in such a way that it never waits for a response from a line of code. An example of this is listed below.

var fs = require('fs');

fs.writeFile('/tmp/1.txt', 'some content that takes long to write', function() {
    console.log('done writing file');
});

console.log('some other message');

The code above does 2 things , it writes to messages to the console and writes a file to the disk. When you run the code because writeFile is non blocking it fire of the text writing code and immediately print ‘here I am’ , once the file is finished writing it will print ‘done writing file’. In a synchronous language we would typically write it something like this :

file.writeText('some text')
print done writing file
print some other message

Because line 2 would only print once the code is finished writing the text to file in contrast if we had to write the same code the done writing file would fire before the file was finished writing to disk

The code below is a example of a node hello world application. You can paste this into the node interpreter or save it to a js file. If executing it from a js file run , node <filename.js> where filename is the name of the file you saved the text into

Hello World Server

var http = require('http');
http.createServer(function (req, res) {
        res.writeHead(200, {'Content-Type': 'text/plain'});
        res.end('Hello World\n');
        }).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');

This example creates and hosts a web server on port 1337 , when ever a client connects to it (with a web browser,curl,wget) it will return the words Hello World. Something to be kept in mind is that this is all non blocking so when the code runs it doesn’t halt and the createServer line and wait for input. Rather it hooks up a callback to the request that will be executed every time a user requests the url. This is important because until a user hits the server the pc is not doing anything it is simply idling and waiting for a request on the web server.

So why would you want to use node over any other language , well there is no magic bullet with node , you simply have to evaluate it to see if it makes sense to use node or not to use it. For example are you happy that its still in development , do you need to be able to handle multiple concurrent connections , what is the size of your project. These are just some of the questions that you need to ask.

If you want a better explanation of the ups and downs of node and how it handles processing I highly suggest reading this article by Felix Geisendorger

So lets build something


We are going to walk through building a fairly simple IRC bot that will log all the conversation in a channel as a means of archiving data.

Before building there are a few things we need to know about what the bot should do.

  1. Connect to IRC and idle
  2. When a user send a message to the channel it should save it to file.

We next need to decide what node.js libraries we want to use , in almost all cases there are already libraries out there for interacting with various services.

  1. There is already a IRC lib called irc so half our work is done.

The file system interaction is built into the default libraries node provides so we dont need worry about anything else. Node.js libraries can be acquired by 2 methods , either get the source code directly from the user( or repo ) or install it with a package manager. There are quite a few package managers out there but we will be using npm.

Getting Started


This tutorial will be run on a mac and should work on any unix enviroment , windows users will just have to adapt.

First follow the instructions here to install node and npm. Only proceed once you have both installed.

Next in your console run the following command to create a dir and change into it and then finally create a blank file that we will fill later.

    mkdir ircWatch && cd ircWatch 
    touch app.js

Next install the libs that we are going to be using with the commands below.

    npm install irc

By default npm install modules into a local folder called node_modules unless you use the -g option to install into the global dir. You will notice that when ever node installs a library it all the source files for that library can be found in the folder ./node_modules/moduleName/lib/. This is very usefull for debugging and generally learning how the various libraries are written.

app.js


Lets start by loading and caching the file system and irc libraries.

    var ircLib = require('irc');
    var fs = require('fs');

This does 2 things it loads the irc lib and stores the resulting library object in the ircLib variable. If you ever need to make calls to the irc library you now do it throught the ircLib variable. We do the exact same thing for the file system library (called fs).

Next we need to setup the irc connection and handle any events it may expose. This is done as follows :

    var client = new ircLib.Client('republiccommandos.co.za', 'mybot', {
        channels: ['#jumpdeck'],
    });

Here we are creating a instance of the irc client and passing some default parameters to it. The first param is the server to connect to , next the name of the bot and finally the channel to join. By default the bot is set to auto join as soon as the app is run ,if you do want this you can add autoConnect : false after the channels and then call client.join('#channel') to join a specific channel. If you want more info about what parameters you can pass and what else the lib can do look at the readme here

You now have the base for the app complete if you save the file , you can start your bot with the command node app.js. This will run the app and connect to the server you specified in your parameters. The bot does nothing yet but we shall give it some power soon.

Logging info and writing to the filesystem


As you may have noticed our bot doesnt do much besides idle , so its time for it to start listening to other people and save it to file. To do this we hook into the events the ircLib exposes. Currently we are interested in the ‘message’ event. To hook up a listener to this event use the following code:

client.addListener('message', function (from, to, message) {
 
}); 

Now when ever the client object fires the message event (which indicates there is a new message in the channel) we can execute our code to save the message to disk.

fs.open('irc.log', 'a', function( e, id ) {
    fs.write( id, message+'\n', null, 'utf8', function(){
        fs.close(id, function(){ });
    });
});

The code above opens the irc.log file in append mode , if the file does not exit it will be created. After its opened it will then write the message the message to file. Its important to note that we need to add a new line char to the end of the file , otherwise all the data will be written to a single line.

Now we dont close the file imediately after calling fs.write since the write is happening async and as such if you close it before it has had a chance to write no data will be saved. To solve this we only call fs.close in the call back function to ensure that its only called after the data is written.

Now you may be wondering at this point if we are going to have inconsistent writes because we are only closing the file after each write. Node handles this correctly by ensuring that the functions fire in the correct order in the background.

    <jameel> hello
    <jameel> test 
    <jameel> 123

So if we had to send the messages above all at once (but in the order above) node will correctly write it to file in the exact same order that it was recived in the channel.

This is a really simple example to get you up and running with something useful in node.js (I spend alot of time in IRC) , but there is alot more that you can do with this example you can add date stamps , save to a db instead of to file, create a file per day to name but a few.

We recently released a tool at http://cc.thinkst.com to capture and collect infosec conference details. We commented on it [here]. One of the cooler components of it, is the ability to view the relationships between speakers/researchers who have collaborated. This post is a quick introduction to the library we used to build our graphs, with enough info to get you up and running in minutes.

As I mentioned, we use ArborJS library which is a a graph visualization library using web workers and the popular jQuery.

The API is really well documented [here] but like most people, I learn best by example (and there are precious few of these). Hopefully this post will fill that niche, and by the end of it you should have a basic understanding of how to use arbor.js in your project.

Our Aim:


End Result

We will be building a simple contrived example as seen in the image above.

Project setup :


Create a new html page and include script references to the following libraries (Download them from the links provided and host locally):

jquery : googleapis
arbor.js : github
graphics.js : github
renderer : github

(Feel free to download the arbor.js repository and investigate the source to poke at its innards.)

Next add a canvas element to the body of the page:

<canvas id="viewport" width="800" height="600"></canvas>

Finally (below your canvas) create a script tag:

<script language="javascript" type="text/javascript"></script>

At this point you are now setup and ready to start creating your own graph in arbor.js.

Your page source should look like this at the moment:

Step1 Page

Getting started - Creating a node:


In the script tag below the canvas element add the following code :

var sys = arbor.ParticleSystem(1000, 400,1);
sys.parameters({gravity:true});

This does 2 things:

  • Initializes the arbor system;
  • Sets up the renderer to use the canvas element we provided;

The particle system tells arbor how much you want things to move around. (If you set the 3rd parameter to 1, nodes will not move around at all unless you drag and drop them). You can play around with values between 0 and 1 to see how flexible or rigid the graph becomes.

Next add :

sys.renderer = Renderer("#viewport") ;

This initializes the renderer which is used by arbor.js to draw the nodes on screen. If you want more advanced shapes or pictures you need to edit the renderer file to provide this functionality. The file that we linked to provides 2 basic shapes the dot and the rectangle to get you started off.

Next add :

var animals = sys.addNode('Animals',{'color':'red','shape':'dot','label':'Animals'});

In the line above we added a node to the graph, this is the first node and will be our parent node (to which all other nodes are connected to).

The first parameter is the node name and is what is used to reference a node at any later stage. The second parameter is a JS object that describes how the node should look.

In this case we are saying the Animals node should be red in color drawn as a dot and have the label ‘Animals’ as its label. If you save the changes and refresh the page you will see the following image on screen.

First Node

Congratulations you have created your first node! At this point your page source should look like this :

Step2 Page

Linking Nodes:


While creating nodes is useful, it is the links between nodes that give these graphs meaning. We will now introduce relationships between the nodes.

Start by adding two more nodes to the graph as seen below :

var dog = sys.addNode('dog',{'color':'green','shape':'dot','label':'dog'});
var cat = sys.addNode('cat',{'color':'blue','shape':'dot','label':'cat'});

We added a cat and dog with different colors in the code above. If you save refresh the page you will see the following image.

Second Node

To indicate that the dog and cat are animals we connect them using edges as seen in the code below.

sys.addEdge(animals, dog);
sys.addEdge(animals, cat);

If you save your changes and refresh you will now see that the 3 nodes are connected by edges. When calling sys.addEdge you are specifying which 2 nodes you want to connect. If you wish to customize how the line is drawn you can edit the renderer to draw custom code.

Third Node

At this point your page source should look like this :

Step3 Page

Getting things done faster and easier


Our methods so far have been very manual and error prone. Arbor provides a method called grafting help automate this.

To use this method you provide arbor with a json object that contains all the nodes and edges that you want to add to screen. Arbor then adds them to the graph correctly.

(In our example we are hardcoding the input data. The graphs on http://cc.thinkst.com achieve this with an ajax call to our backend that returns the data in the correct format) :

Add the following json to the script tag in your page:

var data = {
nodes:{
    animals:{'color':'red','shape':'dot','label':'Animals'},
    dog:{'color':'green','shape':'dot','label':'dog'},
    cat:{'color':'blue','shape':'dot','label':'cat'}
},
edges:{
    animals:{ dog:{}, cat:{} }}
};

As you can see the nodes are created the same way as before with all their data attributes. The edges are represented slightly differently: they are structured in such a way that the parent is at the top and all children are contained within. If it had to be show in a tree it would look like this :

animals                      The Parent node
dog               Child node
cat    Child node

To use this json in the code we need to do the following :

Replace all the code in your script tag with :

var sys = arbor.ParticleSystem(1000, 400,1);
sys.parameters({gravity:true});
sys.renderer = Renderer("#viewport") ;
var data = {
    nodes:{
        animals:{'color':'red','shape':'dot','label':'Animals'},
        dog:{'color':'green','shape':'dot','label':'dog'},
        cat:{'color':'blue','shape':'dot','label':'cat'}
    },
    edges:{
        animals:{ dog:{}, cat:{} }
    }
};
sys.graft(data);

As you can see we have still have to initialize our arbor system and setup the renderer but instead of manually creating each node and linking them with edges we simply call sys.graft with our json object.

Extending Existing data:


Finally it would be nice to hook in more data onto our existing graph as it becomes available. To simulate this we will set a timeout in javascript for 10 seconds , after this our code will be called to add more data to the graph.

Add the code below after your sys.graft(data) :

setTimeout(function(){
    var postLoadData = {
        nodes:{
            joe:{'color':'orange','shape':'dot','label':'joe'},
            fido:{'color':'green','shape':'dot','label':'fido'},
            fluffy:{'color':'blue','shape':'dot','label':'fluffy'}
        },
        edges:{
            dog:{ fido:{} },
            cat:{ fluffy:{} },
            joe:{ fluffy:{},fido:{} }
        }
    };
    sys.graft(postLoadData);
},10000);

What we are doing above is adding 3 new nodes and then setting up new edges for these nodes. We indicate this in the edges section.

If we didn’t have the sys.graft we would have to iterate over every node that’s currently on screen and check if we need to add new nodes and edges to it. If you save and refresh the page you see a image similar to the one below.

Last Node

Finally at this point your page source should look like this :

Final Page

The project itself is worth checking out and goes much deeper. Check out the code they have provided on github.

Intro


I tend to spend alot of time in the #zacon channel mostly because its really cool place to hang out but also because I learn alot from the people there. Sadly I cant be there all the time of every single day. A little while back I noticed that I was missing really interesting URLs that people has posted while I was away, now while I could look back through the logs for the previous day/night, this started to become a little tedious. Also it doesnt help that other people also had this problem.

So the solution I present is the zaconURL watcher, a bot written in Node.js that sits in the channel and silently logs all URLs that are pasted. Simple idea with a simple implementation , thankfully with a bit of feedback from guys in the channel I got to add a few more features such as a RSS feeds (Per person or the entire channel) and tagging for each of the URLs.

App Stack


Building this was mostly to learn Node.js and get a app out in a single day for a framework I have never used (This probably shows in the code so please be gentle).

The various libs that were used are listed below.

Bot : Node.js

  • IRC
  • Mongoose

General : MongoDB

Web: Django pymongo

The End


Thats it really there is nothing more that I think can be said. If you need specifics on something feel free to shout , I will drop the code for the bot online sometime soon if anyone cares to check it out.

Also if you feel so inclined check out the site and all data captured so far at urls.runawaycoder.co.za

With great power comes great …


Everyone knows the deal use what you know for good etc. etc.

With that in mind this is going to be a really quick intro on using ssh tunnels.

The reason for the header is this , if you want to do this at your company with out permission don’t do it .People really don’t like it when you disobey the rules and breaching a outgoing firewall can really make some people angry (including me). If you want to do this at home to learn stuff then please read on

This has been talked about alot in the past but this is my post on it.

Why :


There are many times when perimeter firewalls can just get in the way of things and as a admin it might be simpler to just tunnel through them.

Another example would be accessing a server that is not directly connected to the net. This can be done by using the ssh server as a proxy.

Finally its a good security practice to tunnel insecure protocols(vnc is the most common example) over a tunnel so that they are less likely to be sniffed.

How :


The following examples can be used on any OS of choice , all that is need is to have a ssh program of your choosing and a remote ssh server.

I will be running the examples via the command line on MacOS X but if you have access to another linux/unix install this will work in the exact same way.

Punching through firewalls


This is probably the simplest , if all you want to do is simply browse with without being monitored etc.

The following command does this :

ssh rc1140@localhost -D 8080

Of course you need replace localhost with the name of your remote server. This will open up a dynamic port on port 8080 , to use this for browsing or any other application simply set it as a socks proxy in the config for the relevant application (e.g. firefox).

Making internal resources accessible to the outside


Again doing this with ssh is really very simple but requires a bit more explanation.

The Command :

ssh rc1140@remoteserver -R 8080:localhost:80

This creates a remote bind on port 8080 on the remote server that connects back to port 80 on the localhost.Note that the connection needs to remain open at all times if you wish to use the local resources (in this case port 80 on the local machine)

This command is especially useful in a scenario where the outside party cant ssh into your server but you can ssh out. Think of a remote support staff that needs to get access to resources on your network while the vpn is down , while they cant get in. You can most surely get out and setup and temp forward for them to use.

Making remote resources available locally


This is the opposite of the example above.

The Command :

ssh rc1140@remoteserver -L 8800:localhost:80

This creates a bind locally on port 8800 and forwards all calls to the remote port and host , in this case it forwards all requests on port 8800 to port 80 on localhost of the remote server. While this doesn’t accomplish much since you can already ssh into why not just browse to it directly. But you can replace localhost with for example the IP of your home router (assuming you can access it from the remote server).

This is also used in the case where you want to use the tunnel to secure your connection when using something like vnc.

Example :

ssh rc1140@remoteserver -L 5900:localhost:5900

Now instead of connecting to the remote server directly you would instead connect to the localhost (make sure you don’t already have a local vnc server running). This will ensure that all your communication over vnc is secure.

I tried to cover all the important base’s with ssh tunnels but please do let me know if I have missed anything

After what seems like ages I have finally got down to doing stuff on my arduino.

The one thing that I have been meaning to do for ages is to get a LCD I have working , its a simple character based LCD (which has 2 rows I think). This makes doing a hello world app quite simple to do and it actually write out ‘hello world’ :).

I have had most of the components for this project for ages (minus the potentiometer). While I dont have much advice to give on this subject (mostly because I dont want to re write something that is already out there and done extremely well) you can find the equivalent tutorial of what I have done at ladyada’s site.

Here are couple of images I managed to take of the end result 1 2 3

From here I hope to finally incorporate this into my universal remote control so that I know which device I am controlling.