More Features for your Jade Templates

If you've seen Amphibian.com today you might notice the Christmas lights hanging from the header. I wanted to decorate the site a little, but I was worried that I might forget to take them down at the end of December. I think I left the lights up on my house until March last year. It was really cold and I didn't want to go outside.

To prevent this from happening to my website, I decided to add some code which will check the date and automatically display the lights only during the Christmas season. Since the HTML for Amphibian.com is generated by the Jade template system and Express, conditionally including some lights could be accomplished by putting a date check in the template for the comic page. The problem? The check for the Christmas season is a little more complicated than I cared to cram into a Jade template. And Jade/Express doesn't provide any utility libraries by default that help in this situation.

Fortunately, it's rather simple to add your own modules that can be used in your templates.

First, I made a holiday module that I can use to determine if we are currently in the Christmas season.

module.exports = function() {

    return {

        /*
         * Returns a Date object that represents Thanksgiving (US)
         * for this year. The fourth Thursday in November.
         */
        thanksgiving: function() {

            var d = new Date();

            d.setDate(1);
            d.setMonth(10);

            while (d.getDay() != 4) {
                d.setDate(d.getDate()+1);
            }

            d.setDate(d.getDate()+21);

            return d;

        },

        /*
         * This function returns true during the Christmas season.
         * I define that as Thanksgiving through the end of December.
         */
        isChristmasSeason: function() {

            var monthToday = (new Date()).getMonth();
            var dateToday = (new Date()).getDate();

            return (monthToday == 11 || (monthToday == 10 && dateToday >= this.thanksgiving().getDate()));

        }

    };

};

This is a standard Node module that exports an object containing two functions. One returns a Date object for Thanksgiving in the current year. In the United States, Thanksgiving is the 4th Thursday in November, so the function finds the first Thursday and then adds 3 weeks. The second function returns true if the current date falls within the Christmas season. I have personally defined that as Thanksgiving (hence the need for the first function) through the end of December.

Now to enable this module in Jade templates, you just need to add it to the locals object in the Express app. Here's an example of how to do that with this particular module:

var express = require('express');

var app = express();

app.locals.holiday = require('./holiday')();

Any fields on the locals object are available in Jade templates. In my comic page template, I can now do do this:

doctype html
html
 include head
 body
  include header
  if holiday.isChristmasSeason()
   include holidayHeader
  div#comicArea
   include comic
  include footer
  include js

The holiday object can be used anywhere it's legal to use JavaScript expressions in the template. I use it in the if block to see if I need to include the special holiday header.

You can use this technique to add whatever module you want to locals and then use them inside your templates. While Jade is not perfect, this capability adds a lot of flexibility to the framework.

Please continue to enjoy this Christmas season by viewing today's comic, which is, appropriately, about the Christmas season.

Amphibian.com comic for 27 November 2015

Refactoring Express Router Middlewares

This past weekend I went back to an old comic to fix some bad code that it used. It had been bothering me for a long time. I knew it was not great when I made it, but since it was a bit of a rush-job (to capitalize on the sudden popularity of a dress color meme) I just lived with it. But no more! It had to be cleaned-up!

The problem with it was that the comic in question stored data about how people interacted with it, but never persisted that data to disk. Every time I re-started the comic application, the previous data was lost. No problem, I just put in a timer that wrote the data as a JSON string to disk every 30 minutes. When the application starts, it reads the file.

But when I was adding this stuff, it bothered me how all the code for the routes for this comic were scattered about in the main app.js file. They really should have been put in a separate file and configured as a router. Again, no problem. I moved the four routes into a separate router module and included it from app.js.

But then I noticed something else that bothered me. I had a bunch of router middlewares now that all used very repetitious blocks of code. Seemed like it was time to make more improvements.

The code originally looked like this.

//------------ set up routes for /data/*

app.use('/data', dataRoutes({
    express: express,
    auth: ensureAuthenticated,
    dataSource: cfact
}));

//------------ set up routes for /memeGen/*

app.use('/memeGen', memeRoutes({
    express: express,
    config: conf
}));

//------------ set up routes for /fb/*

app.use('/fb', fbRoutes({
    express: express,
    config: conf
}));

//------------ set up routes for /colors/*

app.use('/colors', colorRoutes({
    express: express
}));

//------------ set up routes for /images/*

app.use('/images', imgRoutes({
    express: express,
    auth: ensureAuthenticated,
    dataSource: cfact
}));

All the middlewares were passed a very similar object into their constructor, but it was not always identical. Some needed only a single item, some needed three. But they all overlapped. I decided to create the options object only once and pass it to all the middlewares. They could then use whatever they needed and ignore what they didn't. But instead of copying-and-pasting a block of code whenever I want to add another one, I created an array of their names and a function to set everything up. Now I have this:

var express = require('express');

var routers = ["data", "images", "memeGen", "colors", "fb"];

var app = express();

// ... other stuff ...

(function setupRouters() {

    var opts = {
            express: express,
            auth: ensureAuthenticated,
            dataSource: cfact,
            config: conf
    };

    routers.forEach(function(val) {

        try {
            console.log("loading router [" + val + "] ...");
            var r = require("./routers/" + val);
            app.use("/" + val, r(opts));
        } catch (e) {
            console.error("error loading router [" + val + "] - " + e);
        }

    });

})();

With the array of router names configured, I create and call a function which goes over that list and creates each router. It sets each one up with the path to match its name. It's now very easy for me to add new router middlewares.

I always enjoy a good refactoring. Much more than I enjoy taking a picture of myself (see today's comic).

Amphibian.com comic for 2 September 2015

Serving Your Express App With Encryption

While today's comic has nothing to do with encryption, you can actually view it (or any of my other content) using encrypted web communications. Most people refer to this as SSL, which stands for Secure Socket Layer, but SSL is an older and now non-recommended way of securing web traffic. The current method is called TLS and stands for Transport Layer Security. Doesn't make a whole lot of difference what you call it, it's the "s" in "https" when you're viewing a page with the reasonable assurance that no one is able to spy on your network traffic.

Don't mess with my network traffic!

Before last week, Amphibian.com had no way of delivering web pages in a secure manner. Why would it need to? It's just a web comic. But after my Bitcoin Paywall comic got so insanely popular I started to get people asking why I didn't have encryption enabled. Without it, there is a small chance that someone could inject their own Bitcoin address into the response from my server and take the money you are trying to send to me. So I decided to get a server certificate signed by a real authority (more on that Friday) and enable encryption for my comics.

This of course led me to create Monday's comic in which the frogs are surrounded by danger when viewed via http://amphibian.com/177 but are much safer when viewed via https://amphibian.com/177.

Now for the technical part. In order to get my Node/Express app to serve web pages over both secure and insecure web ports, 443 and 80 respectively, I had to make a few changes.

For simple apps, you are probably familiar with listening on a port this way:

var express = require("express");
var app = express();

// ... set up routes ...

var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

But if you want the same Express instance to handle the traffic on multiple ports, you have to use the Node http module directly to create servers and pass in the Express instance as a parameter:

var express = require("express");
var http = require("http");
var app = express();

// ... set up routes ...

var server = http.createServer(app).listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

You could of course use this technique to listen on multiple insecure ports, like 3000 and 4000 as in this example:

var express = require("express");
var http = require("http");
var app = express();

// ... set up routes ...

var server1 = http.createServer(app).listen(3000, function() {
    console.log("listening on port %d", server1.address().port);
});

var server2 = http.createServer(app).listen(4000, function() {
    console.log("listening on port %d", server2.address().port);
});

However, if you want one of the listening ports to use encrypted communications you need to use the Node https module instead of http for one of them. In the simplest possible configuration it takes just one extra parameter - an options object which contains at a minimum the private key and public certificate for the server.

var express = require("express");
var http = require("http");
var https = require("https");
var fs = require("fs");
var app = express();

// ... set up routes ...

var server1 = http.createServer(app).listen(3000, function() {
    console.log("listening on port %d", server1.address().port);
});

var sslOptions = {
    key: fs.readFileSync("/path/to/private.key"),
    cert: fs.readFileSync("/path/to/server.cert")
};

var server2 = https.createServer(sslOptions, app).listen(4000, function() {
    console.log("listening securely on port %d", server2.address().port);
});

And just like that you can enable encrypted communications in your Express web app. Now read today's comic where the frogs try to avoid upsetting the apple cart...

Amphibian.com comic for 10 June 2015

Making Bitcoin Paywall Middleware

Today's comic doesn't include a lot of built-in technology like Wednesday's did. Just a moth joke. Or was that just a mistake? I'm not sure. Moving on...

Wednesday's comic had a Bitcoin paywall, and in Wednesday's blog I explained how I processed a 402 response in the client to make it work. But the server played more than a bit part in the whole operation. Get it? Bit part. Bitcoin. Moving on...

It's like regular money, but with more math.

To handle the Bitcoin payments and unlock the content, there are several pieces at work. I am using the Express framework for my web application, so I have a middleware applied to each non-free resource. There is also a Bitcoin address generator to create new payment addresses. Another resource serves as the payment notification - when payment is made to an address, this resource receives data about the transaction. Finally, there is a datastore that manages the records to keep all the other pieces in sync.

This is how it works:

First, a resource is requested which requires payment. The middleware checks with the datastore to see if the requesting client has an existing record for this resource.

If no existing record is found, it means the client has never attempted to access this resource before. The middleware uses the address generator to create a new payment address record and gives it to the datastore. The client is returned the 402 response along with the special payment instruction headers.

Alternatively, the datastore might find a record for this resource and client. That means the client has been here before. The record returned by the datastore will indicate whether or not the payment is complete. If the payment has been made, the middleware allows normal access. If payment has not been made, the middleware returns the 402 response to the client with the special payment instruction headers. The data for those headers comes out of the payment record.

At any time, an external Bitcoin processing system can determine that payment has been made to one of the addresses generated by the address generator. It sends data on the payment transaction to the payment callback resource which looks up the record in the datastore and marks it as paid.

I used the Coinbase Merchant API as the external Bitcoin processing system. When I create addresses with Coinbase, I can specify a callback URL for each one and Coinbase will POST some data to it when payment is made. Each address gets a unique URL so my system can look up the payment record that matches it. I am using the official Coinbase Node module in my app.

Here is a basic outline of the app:

var express = require("express");
var app = express();

// mock purchase record
var demoRecord = {
    recordId: "12345",
    code: "product-code",
    paid: false,
    cost: 123.45,
    address: "bitcoinaddress",
    secret: "itsasecret"
};

// mock datastore object
var datastore = {
    findRecord: function(req, productCode, callback) {
        callback("12345");
    },
    checkPaidStatus: function(recordId, callback) {
        callback(demoRecord);
    },
    newRecord: function(data) {
        // store the record
    },
    findRecordBySecret: function(secret, callback) {
        if (secret === demoRecord.secret) {
            callback(demoRecord);
        } else {
            callback(null);
        }
    }
};

function newBitcoinAddress(productCode, callback) {
    callback(null, demoRecord);
}

function send402(res, data) {

    res.setHeader("X-Payment-Types-Accepted", "Bitcoin");
    res.setHeader("X-Payment-Address-Bitcoin", data.address);
    res.setHeader("X-Payment-Amount-Bitcoin", data.cost);
    res.sendStatus(402);

}

function paywallMiddleware(req, res, next) {

    // let's use the URL as the product code,
    // since that's really what is being sold
    var productCode = req.path;

    datastore.findRecord(req, productCode, function(recordId) {

        if (recordId) {

            // found a record, which means that this client
            // has a payment address and purchase record already.
            // now check to see if it has been paid.
            datastore.checkPaidStatus(recordId, function(data) {

                if (data.paid) {

                    next(); // all paid, move along...

                } else {

                    // respond with the payment address that already exists
                    send402(res, data);

                }

            });

        } else {

            // no record found, which means a new Bitcoin
            // address and purchase record must be created.

            newBitcoinAddress(productCode, function(err, data) {

                if (err) {
                    next(err);
                } else {

                    datastore.newRecord(data);
                    send402(res, data);

                }

            });

        }

    });

}

app.get("/free", function(req, res, next) {
    res.send("free content");
});

app.get("/paid", paywallMiddleware, function(req, res, next) {
    res.send("paid content");
});

app.get("/callback", function(req, res, next) {

    var secretCode = req.query.secret;
    datastore.findRecordBySecret(secretCode, function(record) {
        if (record) {
            record.paid = true;
        }
    });
    res.sendStatus(200);

});

// ------------ start listening
var server = app.listen(3000, function() {
    console.log("listening on port %d", server.address().port);
});

That looks like a lot of code, but it's not really that complicated. Near the top, you see I have some mock objects for the demo. In a real scenario, your datastore object would access an RDBMS or the file system to store and retrieve actual purchase records. Here, though, a single fake record and some stub functions will suffice. The newBitcoinAddress function is also a stub - in the real app, this is where I use the Coinbase API to create addresses and associated purchase records each with their own unique secret code as part of the callback URL. That secret code should never be returned to the user, but needs to be stored so the record can be retrieved later in the callback handler. The send402 function is just a utility function that sets the special response headers and sends the 402 code.

The paywallMiddleware function is the implementation of the algorithm I described above. It uses the mock datastore and address generation function.

The routes at the bottom allow testing the system. Launch the app and point your browser to http://localhost:3000/free. You should get the free content. Now try http://localhost:3000/paid. You should get a 402 response. This is because the /paid route includes the paywallMiddleware. To get access, you'll need to pay.

This demo doesn't require any real Bitcoin transactions to take place. To simulate the callback coming from Coinbase, I included a /callback route that is a GET, which makes it easy to test from your browser. In real life, this will be a POST. To simulate sending payment to the address, you just have to go to http://localhost:3000/callback?secret=itsasecret. When you do, you should get a simple OK response...but on the server side, the demo record was updated to paid because the secret code matched. Now you can go back to http://localhost:3000/paid and get the paid content. If you had used a different (incorrect) value for secret, the /paid resource would not have been unlocked.

Next week I'll go into a little more detail about how I use the Coinbase API. My hope is that someday more content creators such as myself can use functions like this for micropayments in place of running advertisements on their sites.

But that's just a dream right now. How long will it take to become reality? A few moths? I mean, months?

Amphibian.com comic for 22 May 2015

The Origin of Toads.co

Today's comic got a little complicated. So some sinister-looking toad has started a new company to compete with Amphibian.com and has hired away all the worker frogs. Since everybody's over at the other company today, the comic is actually at the toad's company website - toads.co.

I thought it would be simple to point toads.co to the same server as amphibian.com and have the application just use a different stylesheet when the toads URL is detected. As it turns out, it wasn't really that simple.

One of the things I forgot about was how picky browsers can be about cross-domain loading. While there's no issue with sourcing images or JavaScript from other domains, I ran into a minor issue with Font Awesome loading font files cross-domain.

If you haven't heard of Font Awesome, you should check it out. By creating a custom font of pictures (remember Wingdings? It's like that, but useful) it essentially gives you a library of scalable vector icons that can be customized with CSS.

Due to the way Cross-Origin Resource Sharing is designed, a server is expect to specify in a response header if the requesting domain is allowed to use a web font. The header is called Access-Control-Allow-Origin. If amphibian.com is hosting the font, by default only pages at amphibian.com can use it. But if I add the special header to the response when I serve the font, I can specify other domains that are allowed to access it.

My goal, therefore, was to set the "Access-Control-Allow-Origin" header to "http://toads.co" so my social media icons wouldn't be missing when viewed from toads.co. Initially I wasn't sure how to do that, since all static content is served automatically by Express's static middleware. This is the only middleware that is still "built-in" to Express, but it is really just a wrapper for the serve-static module.

To add a certain response header to only some files, I had to supply a setHeaders function when I set up the middleware.

app.use(express.static("public", {
    "setHeaders": function(res, path, stat) {
        if (path.match(/\.(ttf|ttc|otf|eot|woff|font.css|css)$/)) {
            res.setHeader("Access-Control-Allow-Origin", "http://toads.co");
        }
    }
}));

Personally, I think the name of the function is a little misleading. While the intention is certainly to give you a place to set headers, you can pretty much do whatever you want inside. You get access to the response object, the path of the request, and the file stat for the file at that path. There's a small caveat - because the function doesn't return anything, you can't really make any asynchronous calls that defer any processing. You have to do whatever you need to do synchronously. Set the headers you need to set and move on.

My method simply checks to see if the requested file ends with one of the font extensions, and if it does the special header gets set.

I set out to do something that I thought would be simple, and ended up learning something new. Not a bad deal. I hope those frogs got as good of a deal at their new jobs...

Amphibian.com comic for 21 January 2015

Call Now, Log Later

Sometimes, you really want to cut down trees. But most of the time, logging is just having your software write information to a location where you can examine it at a later time. If you're looking for advice on chainsaws or flannel shirts, I'm sorry but this isn't the blog for you.

I can't tell you much about forestry, but I can tell you about logging from your Node.js application using the Express framework.

Logging, pre-Internet.
AZ Lumber & Timber Co. - 1937

Often, using Express means using the Morgan middleware for logging. Morgan is the HTTP request logger for Express. It allows you to easily output all the typical information you might want to see about your requests and responses. It's easy to use and allows you to customize the format or select one of several standard types.

But what if you want to log differently? I wanted to write the typical HTTP request/response data to a database table so that I could run queries against it later. For testing purposes, I also wanted to be able to optionally output the logs to the screen instead of the database.

I ended up writing my own logger middleware which can use either Morgan (for output to the screen) or a custom function (for output to my database) depending on a flag in my configuration object.

Take a look...

module.exports = function(conf) {

    var myLogger = null;

    if (conf.logToDatabase) {

        // here is where you can set up your database
        // connection or whatever you need

        myLogger = function(req, res, next) {
   
            function storeData() {

                res.removeListener('finish', storeData);
                res.removeListener('close', storeData);

                var reqIp = req.ip;
                var reqVerb = req.method;
                var reqPath = req.originalUrl || req.url;
                var resStatus = res.statusCode;
                var resSize = res.get('content-length');
                var reqRef = req.get('referer');
                var reqAgent = req.get('user-agent');

                // here is where you would call the function to
                // insert your data into the database

            };

            // defer logging until the end so we know the response stuff
            res.on('finish', storeData);
            res.on('close', storeData);   

            next();

        };

    } else {

        var morgan = require("morgan");
        myLogger = morgan("combined");

    }

    return function(req, res, next) {
        myLogger(req, res, next);
    };

};

Based on a check of the logToDatabase field in the supplied conf object (line 5), I decide which logger function to wrap in my returned function. If logToDatabase is defined and true, I create my own. Otherwise, I create an instance of Morgan.

Keep in mind that logging middleware is usually specified first in your Express application. That means the logger function is going to be called before any of your other routes get their chance to process the request. How can the log include things like the response code and content-length if those things haven't been determined yet? The trick to getting the HTTP response data into the logs is found on lines 31 and 32.

Notice that my custom logger doesn't actually do much when it is invoked. It defines a function, storeData, but doesn't actually call it. Instead, I assign the storeData function to be a listener on the "finish" and "close" response events and then just call next(). After that, all the other routes can process the request. Eventually, everybody is done with it and the Express framework ensures that at least one of the "finish" and "close" events gets fired.

It's then that storeData gets invoked. Thanks to the closure, it still has access to the request and response objects - and the response object will be fully populated with data.

On lines 14 and 15, you'll see that I remove the listener from both events. This is because I don't know which of the two events got fired, and it is possible that both will be fired. I don't want to log things twice! Since I can be certain that the function has been called at this point (since it's inside the function!) it is safe to remove both listeners.

Call now, log later.

This is a very useful pattern when developing for Node, not just when logging HTTP request data with Express. Creating function closures and using them to defer execution until an event occurs is a common theme that you'll encounter again and again. Now you'll be prepared.

Amphibian.com comic for 7 November 2014

Socket.io with Express

Today I introduced the world to 2-player Solitaire. It was the joke in today's comic, but I thought that it might work in real life too. At the very least, it helps add dimension to the joke.

In it, you and a friend are looking at the same card table. For each valid card placement, you get 1 point. Your goal is to put the cards on valid placements before your opponent does and thereby score the most points.

I've been working on it for about 3 weeks now, which is why I haven't had time to put much code in this blog. But now that it's ready (don't hesitate to send a bug report if you find anything weird) I have time to talk about different parts of it on a technical level.

It is a web application, using just HTML5 and JavaScript. It is playable on both mobile devices and your desktop. Since the main premise is a shared game board, I used Node and Socket.io to share the real-time game data between two web browser clients. I've been using Socket.io for years now, but this was only the second time I've used the 1.0 release combined with the Express framework. The first time I struggled with some things, so I thought I'd share the basic application outline that I came up with to make your life easier.

I'm using Express 4.8.7 and Socket.io 1.0.6 in this project. To set up your application to handle both "normal" HTTP traffic and the various Socket.io communication types, you have to deviate slightly from a typical Node/Express setup.

var express = require("express");
var http = require("http");

var app = express();
var server = http.Server(app);
var io = require("socket.io")(server);

//------------ Socket.io stuff

io.on("connection", function(socket) {
 
    // set up handlers for different events here
    socket.on("some_event", function(data) {
        // do something
    });

});

//------------ web application routes

app.get("/some/path", function(req, res, next) {
 
    res.send("some response");
 
});

// ------------ start the listening

var srv = server.listen(5000, function() {
    console.log("listening on port %d", srv.address().port);
});

The main difference between this configuration and a more typical Express app is what you see on line 4. You have to require the "http" module explicitly, and use it to create a Server instance using your Express app instance. You then pass that Server to Socket.io as I do on line 6.

After that, use your io and app objects to set up your application behavior via event handlers and routes. At the end, start your Server listening. Note that on line 29, it is the http server instance you use to start the listening instead of the Express app object.

Try that, and you're off to a good start. As always, the source code for the entire game is available on GitHub. Check it out if you want to see how I did everything else, or keep reading here and I'll talk about other parts of it soon.

Amphibian.com comic for 22 October 2014

Express Routers as Middleware

Lately I've been developing my web applications in JavaScript, using Node with the Express framework. It's easy to get something thrown together and working very quickly, but like anything else, if you want to maintain it for long it should be well-organized.

Using Routers as Middleware is nice way of partitioning your applications into smaller, easier-to-handle modules that can even be re-used in other projects.

Sometimes you end up with a lot of different routes that have similar prefixes. In my comic application I have things such as

GET  /images
GET  /images/<image>
POST /images
GET  /data/<id>
PUT  /data/<id>
POST /data

...mixed in with a bunch of other routes. All of the /data/* routes are related, and all of the /images/* routes are related. It would be nice to extract them out into their own modules.

We can do this by creating Routers for each set of related paths and using them as Middleware in the main application.

Creating a Router is simple.

var express = require('express');
var myRouter = express.Router();

But how do you use it for something? The Router is a Middleware itself so you use it the same way you use any other Middleware.

var express = require('express');
var app = express();

var myRouter = express.Router();
app.use('/path', myRouter);

Of course, that Router won't really do anything. Let's add a path /cheese that we can GET. Here is a complete app.

var express = require('express');
var app = express();

var myRouter = express.Router();
myRouter.get('/cheese', function(req, res, next) {
    res.send(200, 'cheese is good');
});

app.use('/path', myRouter);

var server = app.listen(3000, function() {
    console.log('listening on port %d', server.address().port);
});

Now we're making progress. If you request /path/cheese you'll get the response "cheese is good." Which it certainly is.

That's great and all, but if you want to really perform some good modularization, the definition of the Router's unique functions should be moved into a separate file. So we would make a new file, cheese.js, that would look something like this.

module.exports = function(express) {

    var myRouter = express.Router();
    myRouter.get('/cheese', function(req, res, next) {
        res.send(200, 'cheese is good');
    });

    return myRouter;

};

And then your main app.js would look like this.

var express = require('express');
var cheese = require('./cheese');
var app = express();

app.use('/path', cheese(express));

var server = app.listen(3000, function() {
    console.log('listening on port %d', server.address().port);
});

And a request for /path/cheese should still tell you that cheese is good.

That's great and all, but here is the real benefit. You can add the same Router for multiple paths. Obviously, you could also reuse your cheese.js module in other applications as well. In the following example, I add the cheese router to both /path and /goat.

var express = require('express');
var cheese = require('./cheese');
var app = express();

var cheeseRouter = cheese(express);

app.use('/path', cheeseRouter);
app.use('/goat', cheeseRouter);

var server = app.listen(3000, function() {
    console.log('listening on port %d', server.address().port);
});

Now you should be able to GET /path/cheese and /goat/cheese. Goat cheese tastes much better than path cheese. Oh, what is path cheese? Well I'm sure you're familiar with leaving a trail of breadcrumbs. If you leave yourself a trail of bits of cheese you have path cheese.

My cheese example is highly simplified but if your module had a number of GETs and POSTs and stuff at different paths, the benefit of breaking it out into a separate file becomes much more clear. It also helps when some routes might need to import another module via require and no other routes need that module. Everything stays neat and self-contained.

Much easier to maintain.

Amphibian.com comic for September 12, 2014

Late for the (Multi)party

Last month I started making a legitimate web application with Node and Express. It's been a generally positive experience, but because I'm using the newest version of Express (4.1) I'm finding a lot of outdated information on the web about how things are accomplished. In version 4.x, Express no longer depends on Connect and most of the previously included middleware is now in separate repos. Overall, I think this is a positive change that adds more flexibility to the system.

Today I wanted to add the ability to perform file uploads to my application. I needed a way to handle the "multipart/form-data" POSTs, and most of the common examples point to the bodyParser middleware. This middleware is no longer recommended for use nor is it included by default (since 4.x) and searching for it led me to a different solution that was more appropriate for my particular needs.

There is a little confusion out there about body parsing middleware. Prior versions of Express included a middleware called "bodyParser" which could handle JSON, urlencoded, and raw request bodies. It is deprecated now in favor of the similarly-named "body-parser" middleware. So what's different about them, beside a hyphen and capitalization?

The new one (hyphenated) does not do multipart posts, which is a good thing. The old one (non hyphenated) did, but did so by writing all the uploaded files to a temp directory and making you process them from there manually. It was easy to forget to clean up the files, and then you might fill up your disk.

Bottom line: don't use bodyParser. Use body-parser. Even if you aren't using Express 4.x, get the new one and use it instead.

But now back to my actual problem. I can use body-parser from now until the cows come home and I still can't process multipart file uploads. There are other middlewares that I could use to process the uploads, so I had to be sure to pick the most appropriate one. In particular, I was looking for one which would allow me to process the stream of the POST body to get Buffer objects which I could write to my database.

I like to party!

The solution that I liked best turned out to not be a middleware at all. After much research, I decided to use Multiparty directly. Multiparty is a module for parsing multipart-form data requests which supports streams2, and doesn't require the data to be written to disk (it can, though, if you really want it to).

Here is a stripped-down version of the code I used, which keeps the uploaded files completely in memory so they can be written to a database or put on your file system somewhere. Whatever you want to do.

var express = require('express');
var multiparty = require('multiparty');

var app = express();

app.post('/images', function(req, res, next) {
 
 var uploadName = '';
 var uploadType = '';
 var chunks = [];
 var totalLength = 0;

 var form = new multiparty.Form();
 
 form.on('error', function(err) {
  // maybe handle errors here, or whatever
  next(err);
 });
 
 form.on('close', function() {
  
  var b = Buffer.concat(chunks, totalLength);
  console.log('storing file %s (%d bytes)', uploadName, b.length);
  // store the image in your database or whatever
  res.send(200);

 });

 form.on('part', function(part) {
  
  part.on('data', function(chunk) {
     chunks.push(chunk);
     totalLength += chunk.length;
  });
  part.on('end', function() {
     uploadName = part.filename;
     uploadType = part.headers['content-type'];
  });
  
 });
 
 form.parse(req);
    
});

var server = app.listen(3000, function() {
 console.log('listening on port %d', server.address().port);
});

It is fairly simple to use in this case. You just create a Multiparty Form object and parse the request with it. It will emit events as it goes.

The "part" event will be emitted for each part of the post. My posts only contain a single form parameter, which is the file, which makes this example really simple. The parameter passed to the handler for the "part" events is a Readable Stream, so it also will emit events as you process the data. I listen for the "data" events and append the chunks to an array of Buffers. When I see the "end" event, that part is over and I set the uploaded file's name and mime-type.

Typically (assuming no errors) the next event emitted by the form is the "close" event. This is where I concatenate all those Buffer chunks together to get the Buffer for the whole upload. I then store the file and return the response code. You have to make sure the listen for the "error" event just in case...if you don't and an error is encountered, your clients will hang forever waiting on a response.

If you want to handle multiple files in a single post, you'd want to move the chunks and totalLength variables into the "part" handler for the form and the processing of the Buffer into the handler for the "end" event for the part instead of the "close" event for the form like I did above. You still have to respond to the client in the "close" handler - don't forget that! It would look something like this:

 form.on('close', function() {
   res.send(200);
 });

 form.on('part', function(part) {

   var chunks = [];
   var totalLength = 0;

   part.on('data', function(chunk) {
      chunks.push(chunk);
      totalLength += chunk.length;
   });
   part.on('end', function() {
      var uploadName = part.filename;
      var uploadType = part.headers['content-type'];
      var b = Buffer.concat(chunks, totalLength);
      console.log('storing file %s (%d bytes)', uploadName, b.length);
      // store the image in your database or whatever
   });
  
 });
 

If you don't like my example, you can also use Multiparty much like the old bodyParser middleware to write the uploaded files to a temp directory if you want. See the readme for the complete API.

I would definitely recommend taking a look at this module if you desire to have a similar file-upload functionality.

Conditional Attributes with Jade

I've been working with Node lately to create a complete web application (not just a Websockets back-end to an HTML5 game) and I am using the Jade template engine with Express.

I really like Express and Jade but the other day I ran into an issue and I just couldn't figure out how to get the HTML output I wanted. The problem was that I wanted to output attributes conditionally. Not just the values of the attributes conditionally, but the whole attribute. Sometimes I wanted to output this:

<img something="whatnot" foo="bar" src="/images/frog.svg" />

and sometimes I wanted to output this

<img something="whatnot" src="/images/frog.svg" />

based on a condition.

It's easy to output attribute values based on conditions. The value for an attribute can actually be any JavaScript expression. So these are all valid in Jade templates:

img(foo='bar')

img(foo=(x == 12 ? 'bar' : 'ribbit'))

img(foo='var-in-the-' + b + '-middle')

I just couldn't figure out how to prevent an attribute from being output at all based on a condition. I finally settled on this:

img(foo=(x == 12 ? 'bar' : ''))

which output this HTML when x is anything but 12:

<img foo="" />

Not really ideal, but I could make it work. I did make it work, but it bothered me. I lost sleep over it. I couldn't eat. I was withdrawn from my family. My children thought that I didn't love them anymore. Okay, maybe I'm exaggerating a little. But seriously, I didn't like it. I didn't want that attribute to come out at all. Today, I found the answer:

img(foo=(x == 12 ? 'bar' : undefined))

Undefined! Of course! Like I said, any valid JavaScript expression can be used. Now when x is anything but 12, the foo attribute is undefined and does not output to the HTML. In case you are curious, null also works in place of undefined. Using undefined is probably better though.

I am sleeping much better now.