Showing posts with label node. Show all posts
Showing posts with label node. Show all posts

Friday, November 27, 2015

More Features for your Jade Templates

If you've seen Amphibian.com today you might notice the Christmas lights hanging from the header. I wanted to decorate the site a little, but I was worried that I might forget to take them down at the end of December. I think I left the lights up on my house until March last year. It was really cold and I didn't want to go outside.


To prevent this from happening to my website, I decided to add some code which will check the date and automatically display the lights only during the Christmas season. Since the HTML for Amphibian.com is generated by the Jade template system and Express, conditionally including some lights could be accomplished by putting a date check in the template for the comic page. The problem? The check for the Christmas season is a little more complicated than I cared to cram into a Jade template. And Jade/Express doesn't provide any utility libraries by default that help in this situation.

Fortunately, it's rather simple to add your own modules that can be used in your templates.

First, I made a holiday module that I can use to determine if we are currently in the Christmas season.

module.exports = function() {

    return {

        /*
         * Returns a Date object that represents Thanksgiving (US)
         * for this year. The fourth Thursday in November.
         */
        thanksgiving: function() {

            var d = new Date();

            d.setDate(1);
            d.setMonth(10);

            while (d.getDay() != 4) {
                d.setDate(d.getDate()+1);
            }

            d.setDate(d.getDate()+21);

            return d;

        },

        /*
         * This function returns true during the Christmas season.
         * I define that as Thanksgiving through the end of December.
         */
        isChristmasSeason: function() {

            var monthToday = (new Date()).getMonth();
            var dateToday = (new Date()).getDate();

            return (monthToday == 11 || (monthToday == 10 && dateToday >= this.thanksgiving().getDate()));

        }

    };

};

This is a standard Node module that exports an object containing two functions. One returns a Date object for Thanksgiving in the current year. In the United States, Thanksgiving is the 4th Thursday in November, so the function finds the first Thursday and then adds 3 weeks. The second function returns true if the current date falls within the Christmas season. I have personally defined that as Thanksgiving (hence the need for the first function) through the end of December.

Now to enable this module in Jade templates, you just need to add it to the locals object in the Express app. Here's an example of how to do that with this particular module:

var express = require('express');

var app = express();

app.locals.holiday = require('./holiday')();

Any fields on the locals object are available in Jade templates. In my comic page template, I can now do do this:

doctype html
html
 include head
 body
  include header
  if holiday.isChristmasSeason()
   include holidayHeader
  div#comicArea
   include comic
  include footer
  include js

The holiday object can be used anywhere it's legal to use JavaScript expressions in the template. I use it in the if block to see if I need to include the special holiday header.

You can use this technique to add whatever module you want to locals and then use them inside your templates. While Jade is not perfect, this capability adds a lot of flexibility to the framework.

Please continue to enjoy this Christmas season by viewing today's comic, which is, appropriately, about the Christmas season.

Amphibian.com comic for 27 November 2015

Friday, October 16, 2015

JavaScript Collections - Map

I still can't fold them back up!
Ready to be confused? Okay, so you know that JavaScript has a map function on Array objects. It creates a new Array by applying a given function to each element of the Array. But now it also has a Map collection object. Now when you talk about a Map in JavaScript, are you talking about a function on an Array or a key/value pair data structure?

Those of us who did a lot of programming in other languages before JavaScript are probably familiar with classic data structures like the Map. A Map is a (typically) indexed collection of keys and their associated values. If you know the key, you can grab the value. C++ has collections objects, as does Java. JavaScript has not had a native Map until recently, but I've never really missed it. Why? Keep reading.

In many of the newest browsers and in Node 4, we can now use Maps. Their functionality should come as no surprise to anyone familiar with basic data structures found in other languages.

var m = new Map();

m.set("key1", "value1");
m.set("key2", "value2");

console.log(m.size); // 2

console.log(m.get("key1")); // "value1"
console.log(m.get("key3")); // undefined
console.log(m.has("key2")); // true
console.log(m.has("key4")); // false

But this is JavaScript! We needn't limit ourselves to just one type of key or value! Check this out:

var p = new Map();

// function as a value...not too strange:

p.set("f1", function(x) { return x*42; });

console.log(p.get("f1")(2)); // 84

// function as a key...a bit odd:

var w = function(z) {
    return z % 3;
};

p.set(w, "weird");

console.log(p.has(w)); // true
console.log(p.get(w)); // weird

Sure, it's perfectly legal code...but why? I know I'm not the world's smartest software engineer, but I cannot think of an example where putting a function as the key in a Map would make any sense. If you can think of one, please comment below.

Now we're having fun with Maps and we're all excited about JavaScript getting this great new data structure. But this is not a whole lot different from what we could always do in JavaScript with objects, since objects behave pretty much like Maps.

var h = {};

h["key1"] = "value1";
h["key2"] = "value2";
h["func1"] = function(t) { return t / 8; };

console.log(h["key1"]);      // value1
console.log(h["key3"]);      // undefined
console.log(h["func1"](16)); // 2
console.log(h.func1(32));    // 4

See? Pretty much the same as my first Map example up at the top. Fine, plain Objects have no size field...so I suppose that's one thing that Maps offer. There are also some differences is how iteration is handled. Consider the following example:

var myMap = new Map();
myMap.set("k1", 33);
myMap.set("k3", 55);

var myObj = {};
myObj["k1"] = 33;
myObj["k3"] = 55;

for (e of myMap) {
    console.log(e);    // the key and value as an array
    console.log(e[0]); // just the key
    console.log(e[1]); // just the value
}

for (e in myObj) {
    console.log(e);        // just the key
    console.log(myObj[e]); // the value
}

Iterating over the Map with for (e of myMap) gives you values for e that are 2-element arrays containing both the key and the value for each Map entry. Iterating over the Object with for (e in myObj) gives you values of e that are just the key.

I'm not going to tell you that JavaScript Maps are the greatest thing since sliced bread, but maybe sliced bread was never that big of a deal anyway. It's not like people didn't have knives. And speaking of bread...

Amphibian.com comic for 16 October 2015

Wednesday, September 23, 2015

Template Strings - New in Node 4

Today on Amphibian.com, the frogs try to get past ad blockers by interleaving the text of ads with the rest of the text on their website. But since their website is their comic about their website (see Monday's topic - meta), there are problems.

Similarly, when you want to put the values of variables or other expressions in your JavaScript strings you probably do something like this:

var str1 = "some static text and " + someVar + " and some more text";

JavaScript Strings, circa 1905
A cool new ES6 feature now available in Node 4 is the Template String, which can make the above code a little nicer.

Template Strings are a new way of creating strings with embedded expressions. These expressions can be simply variables, formulas, or function calls. To make a Template String, use the back-tick ` (the thing that shares the key with ~ on an American English keyboard) instead of the regular tick ' or double-quote ". Inside, put expressions inside curly braces { } prefixed with a dollar sign $.

Here's an example.

var frogType = "Science";
var numFlies = 42;

console.log("The " + frogType + " frog caught " + numFlies + " flies");
console.log(`The ${frogType} frog caught ${numFlies} flies`);

Both messages print out exactly the same, but the second one, using a Template String, is shorter and looks nicer. I think it helps keep your code cleaner.

Another feature of Template Strings is that they can include literal new-lines. Look at this:

var str1 = "this is string 1\n" +
           "with two lines";

var str2 = `this is string 1
with two lines`;

console.log(str1);
console.log(str2);

console.log(str1 === str2); // prints true

The str2 variable is created by simply starting a new line in the middle of the string instead of embedding the \n. Also note that the two strings are equal when compared, despite being created in different ways. Personally, I'm not sold on this particular attribute of Template Strings. I think putting literal new-lines in there hurts the readability of the code, but you might not have a problem with it. Just throwing it out there!

Obviously, using this technique to include variable values in strings is useful. But you can put more complex expressions inside the braces.

var a = 2;
var b = 4;
console.log(`the product is ${a * b}`);

The code above prints the product of a and b by performing the multiplication in the Template String. But please note that the expression is only evaluated once, when the string is created. The following code exemplifies that:

var c = 3;
var d = 7;
var str3 = `the sum is ${c + d}`;

console.log(str3); // the sum is 10

d = 14;

console.log(str3); // the sum is 10

Changing the value of d after str3 has been created does not change the value of str3. Be careful, however, because the process of creating a Template String can change the values of your variables - just like if you did things the old-fashioned way.

var e = 1;
console.log(`the variable e = ${e++}`);

console.log(e); // prints 2

In the above code, printing out the Template String changed the value of e. Maybe this is what you want...maybe it's not. Be mindful.

But wait! There's more!

Template Strings also support tagging, meaning that they can be passed to a function which can modify their output. This feature is a bit unusual.

To use tagging, simply define a string processor function which should expect to receive as arguments
  1. a string array containing the "normal" parts of the Template String
  2. zero or more additional arguments, each representing an expression value
The tagging function should return a string, which is probably going to be some version of the data passed to it, but doesn't have to be related at all! Here's an example.

var test1 = 33;
var test2 = 44;

function tag(strings) {

    // print the static parts, in order
    strings.forEach(function(s) {
        console.log(s);
    });

    // print the expression values, if any
    if (arguments.length > 1) {
        for (var i = 1; i < arguments.length; i++) {
            console.log(arguments[i]);
        }
    } else {
        console.log("no expressions!");
    }

    // this will be the tagged string value
    return "not the string you're looking for";

}

console.log(tag `Part 1 ${test1} part 2 ${test2}`);
console.log(tag `what? no expressions?`);

I've called my tagging function tag, but it can be anything you want. This first example just prints out the parts of the Template String so you can see how it works. It totally ignores the real contents of the Template String and returns nonsense. Those two calls to console.log both print the return value from tag. Probably not useful, but shows you what can be done. I honestly haven't thought of a legitimate use for tagging yet, but I might someday. Maybe you already know of one?

And there you have it. Template Strings, the good and the weird. I think the hardest part will be hitting that back-tick key. Not used to typing that one...

Amphibian.com comic for 23 September 2015

Monday, September 21, 2015

Symbols - New in Node 4

In continuing my discussions of new ES6 language features now available in Node 4, today I'm turning my attention to Symbols.

I'm not as sold on Symbols as I am on some of the other new features. What is a Symbol? A technical explanation is that Symbols are unique and immutable data types. Their purpose is to provide another way of making private properties on objects.

Ecmascript 6: The Future is Now!

Making Symbols is easy.

var sym1 = Symbol();
var sym2 = Symbol("whatnot");
var sym3 = Symbol("whatnot");

console.log(sym2 == sym3);  // false
console.log(sym2 === sym3); // false

The string parameter passed to Symbol() is optional and simply serves as a label. It doesn't mean anything to the Symbol itself. Making another Symbol with the same label doesn't mean the two Symbols are equal. Each Symbol created is unique.

So what does one do with these Symbols? I said that they're used for making private properties on objects, but how is that accomplished?

Let's start by defining the problem. When you have an object, JavaScript provides no protection of its properties. They can be redefined, or even removed! Consider this example.

var Frog = (function() {

    function Frog(type) {
        this.type = type;
    }

    Frog.prototype.getType = function() {
        return this.type;
    };

    return Frog;

}());

var f = new Frog("Science");
console.log(f.getType()); // prints Science

f.type = 27;
console.log(f.getType()); // prints 27

delete f.type;
console.log(f.getType()); // prints undefined

How can the frog type be made immutable outside of its own functions? Symbols are one possibility. Look at this code, slightly modified to use Symbols.

var Frog = (function() {

    var typeSymbol = Symbol("type");

    function Frog(type) {
        this[typeSymbol] = type;
    }

    Frog.prototype.getType = function() {
        return this[typeSymbol];
    };

    return Frog;

}());

var f = new Frog("CEO");
console.log(f.getType()); // prints CEO

console.log(f.type); // prints undefined

console.log(f["type"]); // prints undefined

var s = Symbol("type");
console.log(f[s]); // prints undefined

delete f.type;
console.log(f.getType()); // prints CEO

Instead of setting and accessing the frog type property internally using the dot notation (this.type), this code uses the bracket notation (this["type"]). BUT it uses the Symbol instead of the string "type" to make it inaccessible from outside. All attempts to read or delete the private type property will be unsuccessful. Even trying to do it with another Symbol fails, because Symbols made with the same label are not equivalent.

So that's Symbols. There are of course other ways to make fields private in JavaScript, such as using more closures or with Object.defineProperty. I haven't experienced any scenario where Symbols are really that much better than the previously available alternatives, but it's always nice to have more options.

I can't think of any way to tie this to the Amphibian.com comic for today, so here's the link to it without further comment.

Amphibian.com comic for 21 September 2015

Friday, September 18, 2015

Block Scoping - New in Node 4.0.0

Today wraps up "games week" on Amphibian.com. I had the frogs playing the lottery on Monday, Let's Make a Deal on Wednesday, and poker today. Since there's not much to discuss concerning poker being a game of skill or a game of chance, I'm back to discussing new JavaScript language features available in the recently released Node 4.0.0. (they've actually released 4.1.0 already but it's mostly just bug fixes).

So let's talk about Block Scoping!


This new ES6 feature gives you a new option for limiting the scope of declared variables to a block, statement, or expression. Just declare a variable with let instead of var and it won't be accessible to the entire function!

Here's an example that illustrates the point.

var x = 37;
if (x > 30) {
    let x = -1;
    let y = 900;
    console.log(x); // prints -1
    console.log(y); // prints 900
}
console.log(x); // prints 37
console.log(y); // prints undefined

if (x > 10) {
    var x = 5;
    var y = 777;
    console.log(x); // prints 5
    console.log(y); // prints 777
}
console.log(x); // prints 5
console.log(y); // prints 777

Inside the first if block, let x = -1 declares a block scoped variable x which is distinct from the global x variable declared on the first line. Changing it inside the block has no effect outside the block. By contrast, the second if block declares var x = 5, which isn't really much of a declaration at all. The x variable already in scope simply has its value reassigned. Once outside the block, it still has the value give it inside the block.

Also note the behavior of the y variable. Declared using let inside the first if block, it is undefined when used outside. But declaring it using var inside the second if block makes it global.

Declaring variables with let also works in for loops.

var i = 50;
for (let i = 0; i < 5; i++) {
    console.log(i); // prints 0 through 4
}
console.log(i); // prints 50

for (var j = 0; j < 5; j++) {
    console.log(j); // prints 0 through 4
}
console.log(j); // prints 5

The first loop, using let, prints the value of the variable i which is scoped to just the for loop. Outside of the loop, i still refers to the global variable. The second loop declares j with var and so j becomes available outside of the loop.

Making good use of let for variable declaration can make your code cleaner and lead to fewer bugs.

One final thing to note, however. While block scoping is available in Node 4.0.0+ it only works in strict mode. That means either launch node with the --use_strict flag or put
"use strict";
at the top of your source file. But you're probably using strict mode already, right?

Good. Now enjoy today's comic.

Amphibian.com comic for 18 September 2015

Friday, September 11, 2015

Generator Functions - New in Node 4.0.0

On Wednesday I wrote about arrow functions, a more concise syntax for anonymous functions that is now available in Node 4.0.0. But the new Node has many more new features! Another of the most interesting and useful additions is generator functions.

Generator functions are special functions which can be exited and then re-entered while maintaining their context. That means their variable bindings are saved across entrances. Their name comes from the ability to use these functions to produce sequences of values, like an Iterator in Java.

Generator functions are declared with the special function* syntax, and contain in their bodies a yield statement. Calling a generator function does not execute the body, but instead returns an iterator object which can be used to invoke the body. The statements in the body will be executed until the yield is reached, at which point control returns to the caller.

Consider this example code, which uses a generator function to print out the first 20 numbers in the Fibonacci sequence.

function* fibonacci(){
    var a = 0;
    var b = 1;
    var c = 1;
    while(true) {
        yield c;
        c = a + b;
        a = b;
        b = c;
    }
}

var gen = fibonacci();

for (var i = 0; i < 20; i++) {
    console.log(gen.next().value);
}

It might not be the most efficient way to generate Fibonacci numbers, but it's a good illustration of a generator. When gen.next() is called for the first time, the fibonacci function body is executed up to the yield statement on line 6. All subsequent calls to gen.next() pick up at the next line (seven), which happens to be inside the while loop. After changing the values of the internal variables, we're back to the yield again. This sequence of steps repeats every time gen.next() is called.

As you may notice, calling .next() actually returns on object with a value property. The value property contains whatever was yielded.

Generator functions can come in handy, but since they are part of the ECMAScript 2015 standard, or ES6, they aren't universally available yet. They've just come to Node. Use in the browser is limited to the latest Chrome and Firefox, which means that you're probably going to want to limit yourself to the server-side if you intend to incorporate them in your code.

But as my frogs have learned, being on the cutting edge can be less dangerous than living in the past.

Amphibian.com comic for 11 September 2015

Wednesday, September 9, 2015

Arrow Functions - New in Node 4.0.0

New! Improved! Skipping Numbers!
In news much more important than whatever my frogs did today, Node 4.0.0 was just released. If you're confused by that version number, you're in good company. I thought we were just on version 0.12.7. Did 1.x.x through 3.x.x end up in the same place as Windows 9?

No, no. Node 4.0.0 represents the merge of the Node and Io.js baselines. Io.js split off Node back in December but the two projects have settled their differences and come back together. This means Node now has support for a bunch of new features from the JavaScript ECMA-262 specification, like promises and arrow functions.

I've talked about promises before, but arrow functions...those are interesting. They are basically an even shorter way of expressing anonymous functions.

Consider this example. Let's say you had an array of the first 10 prime numbers and you wanted to add them all up. In a prior version of Node, you could reduce the array by supplying a simple anonymous function as the argument to reduce.

var arr = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29];

var sum = arr.reduce(function(a, b) {
    return a + b;
});

Not bad. I'm sure we've all done simple anonymous functions like that before. But with an arrow function, the same process looks like this:

var arr = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29];

var sum = arr.reduce((a, b) => a + b);

A bit shorter, and it looks nicer as a single line. So what does that syntax really mean?

The arrow function notation basically says execute one or more statements on the right side of the arrow using one or more arguments from the left side of the arrow. If there is only one statement, the return is implicit (above, the statement a + b is equivalent to return a + b). The arrow, of course, is just an equals sign followed by a greater-than sign. It's really just a more concise way of expressing a function which you may find appealing in certain situations.

// multiple args, one statement
var f = (a, b, c) => c ? a + b : a * b;
f(3, 6, false); // equals 18
f(3, 6, true);  // equals 9

// one arg, multiple statements
var g = a => { var b = a * a; var c = a / 2; return b + c };
g(44); // equals 1958
g(22); // equals 495

Nice, huh? There's a lot more new with Node 4.0.0 as well, so you might want to check it out. I'll probably be migrating Amphibian.com to it within the next few weeks. I expect a painless transition, but we'll see. In the mean time, continue to enjoy the comics served up by a prior version of Node.

Amphibian.com comic for 9 September 2015

Wednesday, August 12, 2015

Using Facebook's Graph API to Count Shares

Have you shared today's comic yet? Getting it shared at least 100 times on Facebook is the only way to unlock the action in the last frame. Well, it also has to be Wednesday. So share it 100 times today or wait until next week to try again.
Bombs don't look like this in real life.
This depiction is illogical.

The joke is that the frogs have discovered a logic bomb in a computer, and logic bombs only go off when their criteria are met. The criteria of this particular one include Facebook shares and the day of the week. Really it's designed to get me lots of attention on social media, but there is a joke in it.

Now to the good part...how does it work? The day-of-the-week part is easy, but checking the number of Facebook shares was slightly more involved. I used the Facebook Graph API.

There is a ton of stuff you can do with it, but for now I'm keeping it simple. I wanted to be able to check how many shares a given URL has. And you ask yourself, "Why doesn't he just look at that little number down below every comic?" Well, I'll tell you. That's a Facebook widget that functions by embedding an iframe in the page. I can't really check what it's doing from my own code. I had to write some server-side code to interface directly with Facebook.

Since share information is public, you can access it just by hitting a Facebook URI with any valid access token.

https://graph.facebook.com/v2.4/?access_token=<your-token>&id=<object-id>

The object id is just the URL that you want to check shares on. For me, it's something like http://amphibian.com/205. The tricky part is getting an access token. The easiest way to do it is to create a Facebook App and use the permanent access token associated with it. Sign up as a Facebook developer, and then create a new web app. After you have a web app set up, go to the Tools & Support menu and select Access Token Tool. You should then see a pair of tokens for your app, a User Token and an App Token. The App Token is the one you want because it never expires.

DO NOT share these tokens with anyone, or put them in client-side code! DO NOT! It will give anyone permission to perform actions as your application. I put the App Token in a server-side config file that is NOT committed to source control (at least not in a public repo).

Once you have both an access token and an object id (of something that's been shared at least once), hit the URL and you should get a JSON response.

Something like this:

{
  "og_object": {
    "id": "1128024717211703",
    "description": "Amphibian.com webcomic for Monday, 10 August 2015",
    "title": "Crowdfunding",
    "type": "website",
    "updated_time": "2015-08-10T08:18:03+0000",
    "url": "http://amphibian.com/205"
  },
  "share": {
    "comment_count": 0,
    "share_count": 12
  },
  "id": "http://amphibian.com/205"
}

I just parse that response and get the number of shares. I am doing this from a Node application, and just like when I used the Imgflip API to make memes, I use the convenient request module:

app.get("/shares/:id", function(req, res, next) {

    var comic = req.params.id;
    var token = "your-token-here";

    request({
        uri : "https://graph.facebook.com/v2.4/",
        qs : {
            access_token : token,
            id : "http://amphibian.com/" + comic
        }
    }, function(error, response, body) {

        if (!error && response.statusCode == 200) {

            var data = JSON.parse(body);

            res.setHeader("Content-Type", "application/json");
            res.send({
                id : data.id,
                shares : data.share.share_count
            });

        } else {
            next(error);
        }

    });

});

Again, I must stress that this code has to be done on the server side, since you don't want to give clients access to your App Token. When the client JavaScript has to check if it's ok to set off the cartoon bomb, I access a URL on my server that essentially proxies the request to Facebook. The code example above sets up a route /shares/:id that will give the share data for any comic to a client without exposing my App Token to the world.

Not too difficult. I just hope my plan is successful and the comic gets lots of shares. Trust me, it's worth it. Go on, check it out. It's also got a Zero Wing reference in there (which you can see before it's shared).

Amphibian.com comic for 12 August 2015

Wednesday, July 8, 2015

Async Made Easy

Asynchronous Code Doesn't Have to be Complicated
One of the best things about functional programming with Node is how simple it can be to perform lots of potentially slow functions asynchronously.

That's also one of the worst things.

It's very easy to write really bad code or code that doesn't act the way you expect if you lose sight of the asynchronicity of it all. Fortunately, there are great utility packages like async that make it simple to write good code that acts properly.

Take the following example. Let's say I have a web service that performs some function to its input and returns a new value. For test purposes, let's use the following Node+Express web application that has a single route which reverses a given string.

var express = require("express");
var app = express();

function reverse(s) {
    var o = "";
    for (var i = s.length - 1; i >= 0; i--) {
        o += s[i];
    }
    return o;
}

app.get("/reverse/:s", function(req, res, next) {

    res.setHeader("Content-Type", "text/plain");
    res.send(reverse(req.params.s));

});

// ------------ start listening
var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

Now if I have another application that wants to call that reversal web service with 1000 strings and wait for them all to complete before moving on, what should I do? Making HTTP calls is an asynchronous operation. I make the call and then supply a callback function to let me know when it finishes. Here's the code that sets up the test: a function that makes the web service call and invokes a callback when complete, and an array of 1000 random test strings.

var http = require('http');

function makeString() {
    var text = "";
    var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
    for( var i=0; i < 25; i++ ) {
        text += possible.charAt(Math.floor(Math.random() * possible.length));
    }
    return text;
}

function reverseIt(str, callback) {

    return http.get({
        host: 'localhost',
        port: 3000,
        path: '/reverse/' + str
    }, function(response) {
        var body = '';
        response.on('data', function(d) {
            body += d;
        });
        response.on('end', function() {
            callback(body);
        });
    });

}

var testStrings = [];
for (var i = 0; i < 1000; i++) {
    testStrings.push(makeString());
}

First look at what is probably the most commonly used - and incorrect - way of making all the web service calls.

// ----- incorrect - no idea when they are all done

for (var j = 0; j < testStrings.length; j++) {

    reverseIt(testStrings[j], function(r) {
        console.log("got: " + r);
    });

}

console.log("finished!");

The problem is that you have no idea when all the calls complete. The console.log("finished!") is output almost immediately after you start the program, and the individual logs of the backwards strings keep coming out. Don't do this.

The thing that people do after realizing that first method fails is to use the callback from the completed web service invocation to call the web service again, and again, and again until all the strings are processed. Something like this:

// ----- does everything in series.
//       really slow, but you know when it's finished.

function rev(idx, callback) {
    reverseIt(testStrings[idx], function(r) {
        console.log("got: " + r);
        if (idx === testStrings.length - 1) {
            callback();
        } else {
            rev(idx+1, callback);
        }
    });
}

console.time("type2");
rev(0, function() {
    console.log("did them all!");
    console.timeEnd("type2");
});

Defining a wrapper function that makes the call to reverseIt and then calls itself in reverseIt's callback does allow you to know when all 1000 calls have completed. But doing one request at a time invalidates all the performance gains made possible by the asynchronous nature of the http calls. The console.time and console.timeEnd functions will be used as proof of how this is the worst of the "correct" methods. In my local tests, the complete set of 1000 string reversals took on average around 675 milliseconds.

What is the better way? We can still make all the web service calls asynchronously if we set up some way of tracking their completion. Something like this:

// ----- better

function revManager(arr, callback) {

    var counter = 0;
    return {
        go: function() {
            for (var i = 0; i < arr.length; i++) {
                reverseIt(arr[i], function(r) {
                    console.log("got: " + r);
                    counter++;
                    if (counter === arr.length) {
                        callback();
                    }
                });
            }
        }
    };

}

console.time("type3");
revManager(testStrings, function() {
    console.log("all done!");
    console.timeEnd("type3");
}).go();

This method creates a manager for reversing all the strings that will only invoke a callback when all have completed. To keep track of how many have finished, it sets up a closure with a counter variable and returns an object containing a function to start the process - the go function on line 7. It loops over the array of strings and calls reverseIt on each one, much like in the incorrect example above. But in this callback function for reverseIt, the counter variable is incremented and then checked to determine if all have finished. If they have, the manager's callback is invoked. That's the point when we are sure that all 1000 have completed. The timing on this method proves that it completes much faster - my tests averaged 203 milliseconds.

But way back up near the top I mentioned how utility packages like async make this easier. So now that I've done it the hard way, take a look at how async makes it easy with a general-purpose each function:

// ----- easy with async

var async = require('async');

console.time("type4");
async.each(testStrings, function(item, cb) {
    reverseIt(item, function(r) {
        console.log("got: " + r);
        cb();
    });
}, function() {
    console.log("complete!");
    console.timeEnd("type4");
});

Async has a bunch of methods for different patterns, but the each function corresponds to the desired behavior in this example. It takes three parameters. The first is the array of items, and second is what they call the iterator function. This function is called and passed each item as well as a callback. The general contract is that the iterator function should do its thing and then call the callback when complete. The third parameter to each is the callback function for when the entire array has been processed.

The timing indicates that async's each function performs exactly the same as my own function, but is much more generic and allows me to write fewer lines of code. That's a win-win!

Follow my example and use the time you save by writing less code to enjoy today's comic!

Amphibian.com comic for 8 July, 2015

Friday, June 26, 2015

Node XML Parsing

XML. Why doesn't it stand for eXtinct Markup Language? It's one of those unavoidable unpleasantries of software development. Sooner or later, you're going to have to deal with it.

Since starting to work with Node, I've been able to ignore XML for quite a while. Most JavaScripty things use the much nicer JSON format for their data. However, this past week I finally had break down and get my hands dirty with XML in Node. It wasn't that bad...

I resorted to this because I wanted to pre-process the SVG images that I upload in the Amphibian.com editor. When I create SVG images with Inkscape, they never have viewBox attributes - which are absolutely necessary for correct display in Internet Explorer. I got tired of adding them manually before uploading and decided that the system should just do it for me. Because the SVG image format is just XML, all I need to do is parse the document and look for the viewBox attribute. If it's not there, I can create it using the values of the width and height attributes.

There are two main ways of dealing with XML. One is using a SAX parser. SAX parsing is basically event-driven and is most useful when you need efficient read-only access to XML documents. The other way is using a DOM parser. DOM parsers build Document Object Models out of the XML and therefore enable random read/write access at the cost of having to store the entire document in memory. Because I need to alter the XML documents, I selected a DOM parsing approach.

I am using the xmldom package. I selected it primarily because it is a native JavaScript implementation. Some of the XML parser packages for Node have library dependencies which make life difficult when you run on multiple platforms like I do.

Here is some example code that does the same thing my web app does - parse a SVG, look for some attributes, potentially add a missing one, and output the SVG. This example reads the SVG from a file instead of from a web form upload.

var xmldom = require("xmldom");
var fs = require("fs");

var DOMParser = xmldom.DOMParser;
var XMLSerializer = xmldom.XMLSerializer;

var svgString = fs.readFileSync("frog.svg", { encoding: "utf8" });

var svgDoc = new DOMParser().parseFromString(svgString);
var root = svgDoc.documentElement;

var svgWidth = root.getAttribute("width");
var svgHeight = root.getAttribute("height");
var svgBox = root.getAttribute("viewBox");

if (svgBox === "") {
    root.setAttribute("viewBox", 
       "0 0 " + svgWidth + " " + svgHeight);
}

console.log(new XMLSerializer().serializeToString(svgDoc));


And this is an example of the SVG XML in the file frog.svg. I cut out most of it to save space here - the only part this example cares about is the root element.

<svg xmlns="http://www.w3.org/2000/svg" width="276" height="281">
    ...
</svg>

Let's break down the steps here. First, require the xmldom object on line 1. That is used to get DOMParser and XMLSerializer objects on lines 4 and 5. The DOMParser is obviously the parser, but the XMLSerializer is what you need if you intend to dump an XML DOM back out to a string.

Line 7 reads the contents of the SVG file into a string. DOMParser MUST have a string to function, NOT a buffer. Specifying the encoding when reading forces the return value from readFileSync to be a string, but other steps may be needed if you are getting the SVG via another mechanism.

Line 9 parses the SVG string into a document. In DOM language, the root node, svg in this case, is known as the documentElement. I set that to a variable named root merely for convenience on line 10.

Lines 12, 13, and 14 are where I read the values of the various attributes. The interesting thing about the xmldom package is how it handles missing elements. Note that on line 16 I have to check to see if the svgBox variable is the empty string instead of null. If the attribute is not present in the document, getting it returns and empty string. I found this to be counter-intuitive as a null would seem to make more sense. But it is what it is...

Line 18 calls setAttribute on the root element to add the viewBox if it was not found. Line 21 uses the XMLSerializer to create a string out of the (possibly modified) SVG DOM and logs it to the screen. If you try it yourself, you should see that the viewBox attribute is in fact added if not present in the original document.

So I processed a little XML with Node and it wasn't a completely terrible experience. It was actually just as nice as any Java XML DOM parsers I've used recently. Probably nicer. The best part is now I can save myself all that time spent manually adding in viewBox attributes to frog pictures. While we're on that subject, read and share today's comic:

Amphibian.com comic for 26 June 2015

Wednesday, June 17, 2015

Optimizing SVG Files with Node

The comics for today and Monday have been scaling jokes, but there's nothing funny about Scalable Vector Graphics. Well, there probably is but I can't think of anything at the moment.

Due to the fact that I have a new baby at home (daughter number 4!) I haven't had as much time to work on code projects, but I have been trying to make some updates to my personal website whenever I have a few minutes. Much like my comics, I want to make use of SVG images of frogs on it. Really big, in-your-face frogs. Frogs that scale perfectly even on retina displays.

The downside to using SVG images is that they can easily become bloated. Sure, if you want a really large image it's almost always better to use an SVG instead of a "normal" image format. And since they are just XML text they compress well. But Inkscape puts a ton of extra stuff in them that doesn't really need to be there - removing all the unnecessary information can make the files even better!

There are some web-based tools for cleaning-up SVG files, but the ones that I tried were not able to process my frogs without destroying them.

But then I found SVGO, the SVG Optimizer for Node.

It is super-configurable, modular, and can be run from the command line or included in an application. I installed and tested it from the command line:
$ npm install -g svgo
$ svgo frog.svg frog.min.svg
This image is the original file, frog.svg.

This one is 15k
...and this image is the optimized file, frog.min.svg.

This one is 5k
See any differences? I don't. And the optimized one is only 5k, whereas the original was 15k.

When I get a few more minutes, I intend to embed it in the workflow of adding or updating images for the comic. Once I do that, all SVGs that I create in Inkscape will be automatically optimized when I upload them before they are stored in the database. This should improve the download times for my comic as well as the render speed since there is less XML for the clients to process.

Not that the comic takes that long to load now...the server does gzip all the files before sending them as long as the client supports it. See for yourself by viewing the comic!

Amphibian.com comic for 17 June 2015

Monday, June 1, 2015

Easy Node Debugging

Today's comic is just silly. A frog's debugger is his tongue, since it is a very effective tool for removing bugs. Or maybe it just relocates them - directly to the frog's stomach. If you want to use a debugger with your Node.js applications, you'll want something better than a tongue.

I will admit that I was slow to adopt the use of a debugger for runtime code inspection. Early on in my software development career, when a program wasn't working the way I thought that it should, I just put in some code to print stuff to the console and restarted. That's when my applications were small and didn't take long to recompile and restart. By 2003, I was working on large Java EE systems where the process of rebuilding and redeploying took too long for System.out.println(...) to be an acceptable method for finding bugs. I actually got pretty good with the Java command-line debugger, but after seeing the Java debugger in the Eclipse IDE I gave up Emacs as my primary development environment and switched. I have no regrets.

Today I also do a lot of JavaScript development for Node in addition to my work with Java, but have been so far unimpressed with the debugging capabilities for Node integrated into Eclipse. I use Nodeclipse and for development purposes I am fairly happy, but the debugger left me wanting something else.

I believe I found it with Node Inspector. The idea behind it is pretty cool - debug your JavaScript applications with the JavaScript debugger with which you are probably most familiar: the Chrome Dev Tools!

Here's how it works. first install Node Inspector as a global module.
$ npm install -g node-inspector
Then instead of launching your Node app the "normal" way, do this instead:
$ node-debug app.js
As long as Chrome is your default browser (come on, why shouldn't it be?), the debugger window will launch as a new browser tab which will look like the Developer Tools window. Awesome!

Node Inspector debugging one of my demo apps. Using the Chrome Dev Tools in Chrome!

There are a few things to note here. First, you'll want to be using a version of node greater than or equal to v0.11.13. Older versions work but the debugging capabilities are limited. Second, since the debugger is also running in the browser, if you are debugging a web app you might find yourself needing to switch tabs all the time. You can pop the tab out into a separate window, but it's still a third thing you need to deal with in addition to your IDE and the application you are testing. Not a huge problem, but it is something to consider. Maybe it's enough to make you finally add that third monitor to your development workstation...

Or if you want to get rid of bugs the old-fashioned way, try a frog.

Amphibian.com comic for 1 June 2015

Monday, May 25, 2015

Using the Coinbase Merchant API with Node

I've been writing about different pieces of my Bitcoin integration since last week when Wednesday's comic broke the Fourth Wall while building a paywall. First I explained the client-side code that handled the 402 response. Then on Friday I gave an outline of my Express middleware that controls the access to non-free resources. I used several stubs and mock objects in that post because the actual implementations of the pieces will vary based on where they are integrated - it's just not very interesting what I did with my database access because it's probably different from yours.

But one last part in which people may be interested is the integration with Coinbase's Merchant API via their Node module. I used Coinbase in my implementation because their API was very easy to work with and the integration points are extremely modular. It is set up in such a way that I could easily replace Coinbase with another service such as BlockCypher or calls to the Blockchain API. But I had a wallet with Coinbase already so I went with that...

There are really only two interactions with Coinbase that I perform. One is creating a new Bitcoin address that is tied to my Coinbase wallet. The other is processing the callback from Coinbase informing my server that payment has been made.

Creating a new address is very simple if you've set up your Coinbase API access correctly. Log in to your Coinbase account and go to https://www.coinbase.com/settings/api to create a new API key. Make sure it has the "address" permission and it is enabled. You have to verify every step with 2-factor authentication so have your phone nearby. If possible, set the IP restriction on your key as an extra precaution against unauthorized access.

Your key has two parts, the key and the secret. After creation, you can copy them and use them with the Coinbase Node module. Keep them secure! Don't put them directly in your code and accidentally push it to a public repo on GitHub! Besides the key and secret, you'll also need the Coinbase account number for which you want to generate addresses. Your account number is basically just a Coinbase identification for each wallet you have with them. It's displayed as part of the URL when you are looking at your wallet on their web site.

Once you have those three things, to generate a new address you do something like this:

var Client = require("coinbase").Client;

var client = new Client({
    apiKey: "your-coinbase-api-key",
    apiSecret: "your-coinbase-api-secret"
});

var accountId = "your-coinbase-account-id";

var Account   = require("coinbase").model.Account;
var myBtcAcct = new Account(client, {
    "id" : accountId
});

var secretCode = "secret-code-unique-to-each-address";

var args = {
    callback_url: "http://example.com/callback?secret=" + secretCode,
    label: "your address label"
};

myBtcAcct.createAddress(args, function(err, data) {

    if (err) {
        console.log("unable to create address: " + err);
    } else {
        console.log(data.address);
    }

});

The basic steps are to create a client, create an account object with that client, and then call createAddress on that account. The createAddress function takes an arguments object as the first parameter, in which you specify an optional callback URL and label. The label is just some text you can add for your own reading later. It has no functional aspect. But the callback URL is very important. As you can see on line 18, I use a secret code as part of the callback query string. In my example it is hard-coded, but in real life you'll want to generate a different one for every address. If you keep the secret code only on your server - never send it to the client in any way - it ensures that someone won't be able to send their own POST to your callback URL and trick your server into thinking that they have paid. Store the address and secret code in your database and make sure they match in your callback processor. If they don't, something is wrong and you shouldn't honor the validity of the callback!

If you wanted, you could take the code above and plug it into the example app from Friday's post to generate real addresses. If you wanted to handle the real callbacks in that app as well, you'd want to be able to process the actual data format that comes from Coinbase with those POSTs. The POST to your callback URL will contain this as the body:

{
  "address": "somebitcoinaddress",
  "amount": 1.234,
  "transaction": {
    "hash": "somebiglonghashvaluething"
  }
}

For me, the most important parts were the secret code in the callback query string, the Bitcoin address from the body, and the amount from the body. I don't need the transaction hash, but other applications might. The two pieces of information from the POST body are easy to extract if you use the body-parser middleware for JSON. Here is an example of an app that just takes the callbacks - you could substitute the callback route from Friday's post with this one if you want.

var express = require("express");
var bodyParser = require('body-parser');

var app = express();
app.use(bodyParser.json());

app.get("/callback", function(req, res, next) {

    var secretCode = req.query.secret;
    var addr = req.body.address;
    if (addr) {

        var amountRecieved = req.body.amount;

        // make sure the secret and address
        // match your records, and if so
        // mark the transaction as paid if
        // the amount >= the price.

    } else {

        console.log("something went wrong!");

    }

    // always give coinbase a 200
    res.sendStatus(200);

});

// ------------ start listening
var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

One thing to note (that I missed at first) is that a customer doesn't have to pay the full amount in a single transaction. If you were charging 0.05 BTC, it would be perfectly fine to get two callbacks, each indicating a payment of 0.025 BTC. When handling the data, make sure you add up amounts sent to the same address instead of just taking the latest.

I think I've covered all the major parts of my paywall function now. Just to be clear, I'm not actually advocating everyone putting up paywalls on all their content. Last week's comic was just a joke that happened to include a novel way to handle online micropayments. I do believe that there are lots of legitimate use cases for Bitcoin microtransactions on the web, particularly for content creators such as myself. Charging a few cents worth of cryptocurrency here and there for a pieces of premium content or special features could be a viable alternative to displaying ads on the web (tipping is another alternative, which I've discussed previously).

So enjoy today's comic - completely free!

Amphibian.com comic for 25 May 2015

Friday, May 22, 2015

Making Bitcoin Paywall Middleware

Today's comic doesn't include a lot of built-in technology like Wednesday's did. Just a moth joke. Or was that just a mistake? I'm not sure. Moving on...

Wednesday's comic had a Bitcoin paywall, and in Wednesday's blog I explained how I processed a 402 response in the client to make it work. But the server played more than a bit part in the whole operation. Get it? Bit part. Bitcoin. Moving on...

It's like regular money, but with more math.
To handle the Bitcoin payments and unlock the content, there are several pieces at work. I am using the Express framework for my web application, so I have a middleware applied to each non-free resource. There is also a Bitcoin address generator to create new payment addresses. Another resource serves as the payment notification - when payment is made to an address, this resource receives data about the transaction. Finally, there is a datastore that manages the records to keep all the other pieces in sync.

This is how it works:

First, a resource is requested which requires payment. The middleware checks with the datastore to see if the requesting client has an existing record for this resource.
If no existing record is found, it means the client has never attempted to access this resource before. The middleware uses the address generator to create a new payment address record and gives it to the datastore. The client is returned the 402 response along with the special payment instruction headers.
Alternatively, the datastore might find a record for this resource and client. That means the client has been here before. The record returned by the datastore will indicate whether or not the payment is complete. If the payment has been made, the middleware allows normal access. If payment has not been made, the middleware returns the 402 response to the client with the special payment instruction headers. The data for those headers comes out of the payment record.
At any time, an external Bitcoin processing system can determine that payment has been made to one of the addresses generated by the address generator. It sends data on the payment transaction to the payment callback resource which looks up the record in the datastore and marks it as paid.

I used the Coinbase Merchant API as the external Bitcoin processing system. When I create addresses with Coinbase, I can specify a callback URL for each one and Coinbase will POST some data to it when payment is made. Each address gets a unique URL so my system can look up the payment record that matches it. I am using the official Coinbase Node module in my app.

Here is a basic outline of the app:

var express = require("express");
var app = express();

// mock purchase record
var demoRecord = {
    recordId: "12345",
    code: "product-code",
    paid: false,
    cost: 123.45,
    address: "bitcoinaddress",
    secret: "itsasecret"
};

// mock datastore object
var datastore = {
    findRecord: function(req, productCode, callback) {
        callback("12345");
    },
    checkPaidStatus: function(recordId, callback) {
        callback(demoRecord);
    },
    newRecord: function(data) {
        // store the record
    },
    findRecordBySecret: function(secret, callback) {
        if (secret === demoRecord.secret) {
            callback(demoRecord);
        } else {
            callback(null);
        }
    }
};

function newBitcoinAddress(productCode, callback) {
    callback(null, demoRecord);
}

function send402(res, data) {

    res.setHeader("X-Payment-Types-Accepted", "Bitcoin");
    res.setHeader("X-Payment-Address-Bitcoin", data.address);
    res.setHeader("X-Payment-Amount-Bitcoin", data.cost);
    res.sendStatus(402);

}

function paywallMiddleware(req, res, next) {

    // let's use the URL as the product code,
    // since that's really what is being sold
    var productCode = req.path;

    datastore.findRecord(req, productCode, function(recordId) {

        if (recordId) {

            // found a record, which means that this client
            // has a payment address and purchase record already.
            // now check to see if it has been paid.
            datastore.checkPaidStatus(recordId, function(data) {

                if (data.paid) {

                    next(); // all paid, move along...

                } else {

                    // respond with the payment address that already exists
                    send402(res, data);

                }

            });

        } else {

            // no record found, which means a new Bitcoin
            // address and purchase record must be created.

            newBitcoinAddress(productCode, function(err, data) {

                if (err) {
                    next(err);
                } else {

                    datastore.newRecord(data);
                    send402(res, data);

                }

            });

        }

    });

}

app.get("/free", function(req, res, next) {
    res.send("free content");
});

app.get("/paid", paywallMiddleware, function(req, res, next) {
    res.send("paid content");
});

app.get("/callback", function(req, res, next) {

    var secretCode = req.query.secret;
    datastore.findRecordBySecret(secretCode, function(record) {
        if (record) {
            record.paid = true;
        }
    });
    res.sendStatus(200);

});

// ------------ start listening
var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

That looks like a lot of code, but it's not really that complicated. Near the top, you see I have some mock objects for the demo. In a real scenario, your datastore object would access an RDBMS or the file system to store and retrieve actual purchase records. Here, though, a single fake record and some stub functions will suffice. The newBitcoinAddress function is also a stub - in the real app, this is where I use the Coinbase API to create addresses and associated purchase records each with their own unique secret code as part of the callback URL. That secret code should never be returned to the user, but needs to be stored so the record can be retrieved later in the callback handler. The send402 function is just a utility function that sets the special response headers and sends the 402 code.

The paywallMiddleware function is the implementation of the algorithm I described above. It uses the mock datastore and address generation function.

The routes at the bottom allow testing the system. Launch the app and point your browser to http://localhost:3000/free. You should get the free content. Now try http://localhost:3000/paid. You should get a 402 response. This is because the /paid route includes the paywallMiddleware. To get access, you'll need to pay.

This demo doesn't require any real Bitcoin transactions to take place. To simulate the callback coming from Coinbase, I included a /callback route that is a GET, which makes it easy to test from your browser. In real life, this will be a POST. To simulate sending payment to the address, you just have to go to http://localhost:3000/callback?secret=itsasecret. When you do, you should get a simple OK response...but on the server side, the demo record was updated to paid because the secret code matched. Now you can go back to http://localhost:3000/paid and get the paid content. If you had used a different (incorrect) value for secret, the /paid resource would not have been unlocked.

Next week I'll go into a little more detail about how I use the Coinbase API. My hope is that someday more content creators such as myself can use functions like this for micropayments in place of running advertisements on their sites.

But that's just a dream right now. How long will it take to become reality? A few moths? I mean, months?

Amphibian.com comic for 22 May 2015

Monday, May 11, 2015

Make Your Own QR Codes with Node

Friday's comic showed the frogs' first Bitcoin address as a QR Code framed and hanging on a tree, much like many stores have their first dollar framed. It was a real Bitcoin address, but sadly no one has sent any money to it yet.

I had a bit of a problem getting that QR Code into the comic. If you haven't been reading this blog for long, you may not know that the comics are made out of SVG images. That means I needed a SVG QR Code. No problem, right? There are a bunch of web sites that will generate QR Codes for you in SVG format (this one, this one, this one). Unfortunately, they create them a way which makes it difficult to embed inside another SVG - like my picture frame. I either ended up with nothing, or an incredibly huge file.

Why not try a different approach?

Next I used one of those free online tools to make a .PNG version of the QR Code. It was a very small file, and I could embed the image inside the SVG of the picture frame. Great! Unlike a "pure" SVG, it will pixilize a little when scaled but it's a bunch of blocks to begin with! No problem!

Unless I want it to show up on iOS devices. Bugger!

For whatever reason, the iOS browser doesn't display the image embedded in the SVG. I was, therefore, forced to return to the SVG approach. I had to do it with two separate files - one for the empty frame and one for the QR Code in it. Not exactly what I wanted, but it works.

There is some good that came out of this, however. In my research, I came upon a really nice Node module for generating your own QR Codes in PNG or SVG format. Why rely on some other website to do it for me? It's called simply qr-image, and it has no other package dependencies.

In just a few lines of code, I was able to add a feature to my website which enables me to make all the QR Codes I want. If you use the Express framework, as I do, you can make a simple app like this, just for QR Code generation:

var express = require("express");
var qr = require("qr-image");
var app = express();

app.get("/qrc", function(req, res, next) {

    var text = req.query.text;
    res.setHeader("Content-Type", "image/png");
    res.send(qr.imageSync(text, {type: "png"}));

});

// ------------ start listening
var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

Launch that app, point your browser to

http://localhost:3000/qrc?text=some+text+here

and you should see a nice QR Code. Put anything you want as the value of the text parameter, such as URLs, Bitcoin addresses, secret codes, etc. If you don't want to type the URL out manually, set up a simple web page form that submits the data. Or create your own custom routes that encode data from another source. The possibilities are endless. Well, maybe not endless. But certainly very numerous.

In today's comic, the frogs are trying to find something else that is endless...

Amphibian.com comic for 11 May 2015

Wednesday, April 29, 2015

Automating Code Deployment with GitHub Webhooks and Node

Do you push a "pull" button?
I've been pushing updates to my webcomic's code repo on GitHub for over a year now. My workflow has gone something like this:
  1. Make code changes.
  2. Test locally.
  3. Push changes to GitHub.
  4. Log on to production server.
  5. Pull changes from GitHub.
I'm trying to automate steps 4 and 5. I'd like to have my production server automatically update itself whenever I push code to the "main" branch of the repository.

This week I've been experimenting with using GitHub's Webhooks to automate my code deployment. Webhooks are a cool idea - set one up on your repo and every time an event occurs (push, pull request, fork, etc.) GitHub will send you some data to a URL of your specification which describes the event. All you need is something to process the data!

I set myself up a simple Node application to process the data using the githubhook package. It was very easy to use.

var githubhook = require("githubhook");
var exec = require("child_process").exec;

var github = githubhook({
    port: 8082,
    secret: "your-secret-code"
});

// listen to push on github on branch master
github.on("push", function(repo, ref, data) {

    var branchName = ref.substring(ref.lastIndexOf("/")+1);
    if (branchName === "master") {

        var repoName = data.repository.name;
        if (repoName === "comic") {

            child = exec("/home/comic/update_comic.sh", function(error, stdout, stderr) {

                if (error !== null) {

                    var e = new Error("exec error: " + error);
                    console.log(e);

                } else {

                    console.log("pulled update for " + repoName);

                }

            });

        }

    }

});

// listen to github push
github.listen();

My application will listen on port 8082 for GitHub events, which will be HTTP POSTed in JSON format (when you select that option later). When setting up, the value for secret (line 6) is important. You'll want that to match the value you'll set up in GitHub. Once you create the object, specify callback functions for the events that interest you and start listening.

I am just looking for the "push" event. When that event happens, I check to make sure the push was to the master branch and that the repo was my comic (I might use this for other repos later). If both of those are true, I exec a script that will pull the updates and restart the app if necessary.

Once you have your app running, you can set up your GitHub repo with the Webhook. Just go to your repository page, and click on the Settings link. Then click on "Webhooks & Services" and then on the "Add webhook" button.

Now specify the URL where you are listening with the app. My URL happens to be http://caseyleonard.com:8082/github/callback. The port is obviously the same one I specified above, but the path of /github/callback is the default used by githubhook (you can change it if you want - see the docs). Set the Content Type to be "application/json" and type in the same value for Secret that you used in the app. If these don't match, GitHub will receive a 403 response code when it sends you data and your listener will not be called. You can also select which events of which you want to be notified. It defaults to just "push" which is fine for my use.

It's simple and it works for me, at least so far. The less time I spend updating the code for the comic, the more time I can spend making the comics themselves. Today is the last of the Canterbury Tales parody, a set which took me a very long time to create!