Sunday, May 4, 2014

Late for the (Multi)party

Last month I started making a legitimate web application with Node and Express. It's been a generally positive experience, but because I'm using the newest version of Express (4.1) I'm finding a lot of outdated information on the web about how things are accomplished. In version 4.x, Express no longer depends on Connect and most of the previously included middleware is now in separate repos. Overall, I think this is a positive change that adds more flexibility to the system.

Today I wanted to add the ability to perform file uploads to my application. I needed a way to handle the "multipart/form-data" POSTs, and most of the common examples point to the bodyParser middleware. This middleware is no longer recommended for use nor is it included by default (since 4.x) and searching for it led me to a different solution that was more appropriate for my particular needs.

There is a little confusion out there about body parsing middleware. Prior versions of Express included a middleware called "bodyParser" which could handle JSON, urlencoded, and raw request bodies. It is deprecated now in favor of the similarly-named "body-parser" middleware. So what's different about them, beside a hyphen and capitalization?

The new one (hyphenated) does not do multipart posts, which is a good thing. The old one (non hyphenated) did, but did so by writing all the uploaded files to a temp directory and making you process them from there manually. It was easy to forget to clean up the files, and then you might fill up your disk.

Bottom line: don't use bodyParser. Use body-parser. Even if you aren't using Express 4.x, get the new one and use it instead.

But now back to my actual problem. I can use body-parser from now until the cows come home and I still can't process multipart file uploads. There are other middlewares that I could use to process the uploads, so I had to be sure to pick the most appropriate one. In particular, I was looking for one which would allow me to process the stream of the POST body to get Buffer objects which I could write to my database.
I like to party!

The solution that I liked best turned out to not be a middleware at all. After much research, I decided to use Multiparty directly. Multiparty is a module for parsing multipart-form data requests which supports streams2, and doesn't require the data to be written to disk (it can, though, if you really want it to).

Here is a stripped-down version of the code I used, which keeps the uploaded files completely in memory so they can be written to a database or put on your file system somewhere. Whatever you want to do.


var express = require('express');
var multiparty = require('multiparty');

var app = express();

app.post('/images', function(req, res, next) {
 
 var uploadName = '';
 var uploadType = '';
 var chunks = [];
 var totalLength = 0;

 var form = new multiparty.Form();
 
 form.on('error', function(err) {
  // maybe handle errors here, or whatever
  next(err);
 });
 
 form.on('close', function() {
  
  var b = Buffer.concat(chunks, totalLength);
  console.log('storing file %s (%d bytes)', uploadName, b.length);
  // store the image in your database or whatever
  res.send(200);

 });

 form.on('part', function(part) {
  
  part.on('data', function(chunk) {
     chunks.push(chunk);
     totalLength += chunk.length;
  });
  part.on('end', function() {
     uploadName = part.filename;
     uploadType = part.headers['content-type'];
  });
  
 });
 
 form.parse(req);
    
});

var server = app.listen(3000, function() {
 console.log('listening on port %d', server.address().port);
});

It is fairly simple to use in this case. You just create a Multiparty Form object and parse the request with it. It will emit events as it goes.

The "part" event will be emitted for each part of the post. My posts only contain a single form parameter, which is the file, which makes this example really simple. The parameter passed to the handler for the "part" events is a Readable Stream, so it also will emit events as you process the data. I listen for the "data" events and append the chunks to an array of Buffers. When I see the "end" event, that part is over and I set the uploaded file's name and mime-type.

Typically (assuming no errors) the next event emitted by the form is the "close" event. This is where I concatenate all those Buffer chunks together to get the Buffer for the whole upload. I then store the file and return the response code. You have to make sure the listen for the "error" event just in case...if you don't and an error is encountered, your clients will hang forever waiting on a response.

If you want to handle multiple files in a single post, you'd want to move the chunks and totalLength variables into the "part" handler for the form and the processing of the Buffer into the handler for the "end" event for the part instead of the "close" event for the form like I did above. You still have to respond to the client in the "close" handler - don't forget that! It would look something like this:

 form.on('close', function() {
   res.send(200);
 });

 form.on('part', function(part) {

   var chunks = [];
   var totalLength = 0;

   part.on('data', function(chunk) {
      chunks.push(chunk);
      totalLength += chunk.length;
   });
   part.on('end', function() {
      var uploadName = part.filename;
      var uploadType = part.headers['content-type'];
      var b = Buffer.concat(chunks, totalLength);
      console.log('storing file %s (%d bytes)', uploadName, b.length);
      // store the image in your database or whatever
   });
  
 });
 
If you don't like my example, you can also use Multiparty much like the old bodyParser middleware to write the uploaded files to a temp directory if you want. See the readme for the complete API.

I would definitely recommend taking a look at this module if you desire to have a similar file-upload functionality.

Sunday, April 27, 2014

Conditional Attributes with Jade

I've been working with Node lately to create a complete web application (not just a Websockets back-end to an HTML5 game) and I am using the Jade template engine with Express.

I really like Express and Jade but the other day I ran into an issue and I just couldn't figure out how to get the HTML output I wanted. The problem was that I wanted to output attributes conditionally. Not just the values of the attributes conditionally, but the whole attribute. Sometimes I wanted to output this:

<img something="whatnot" foo="bar" src="/images/frog.svg" />

and sometimes I wanted to output this

<img something="whatnot" src="/images/frog.svg" />

based on a condition.

It's easy to output attribute values based on conditions. The value for an attribute can actually be any JavaScript expression. So these are all valid in Jade templates:

img(foo='bar')

img(foo=(x == 12 ? 'bar' : 'ribbit'))

img(foo='var-in-the-' + b + '-middle')

I just couldn't figure out how to prevent an attribute from being output at all based on a condition. I finally settled on this:

img(foo=(x == 12 ? 'bar' : ''))

which output this HTML when x is anything but 12:

<img foo="" />

Not really ideal, but I could make it work. I did make it work, but it bothered me. I lost sleep over it. I couldn't eat. I was withdrawn from my family. My children thought that I didn't love them anymore. Okay, maybe I'm exaggerating a little. But seriously, I didn't like it. I didn't want that attribute to come out at all. Today, I found the answer:

img(foo=(x == 12 ? 'bar' : undefined))

Undefined! Of course! Like I said, any valid JavaScript expression can be used. Now when x is anything but 12, the foo attribute is undefined and does not output to the HTML. In case you are curious, null also works in place of undefined. Using undefined is probably better though.

I am sleeping much better now.

Tuesday, December 31, 2013

My GitHub Game-off 2013 Entry

This past November I participated in the GitHub Game-off 2013. Basically it was a month-long game jam where you created a game that could be played in a web browser on the theme of "change." When I first read about it, it was already November 6th so I was a little behind. But since I had some experience with creating HTML5 games using JavaScript and Canvas I thought I'd give it a try.

The theme was "change" which made me immediately think of frogs. In fact, almost everything makes me immediately think of frogs. I decided to make a game about the life-cycle of frogs. Starting as a tadpole, you have to grab food and avoid being eaten by fish. Other tadpoles are also out to get you. It's a hard life for tadpoles. So in my game, you move your tadpole around in a pond trying to catch falling food and grow into a tadpole with legs, a froglet, and finally a full-grown frog. Once you are a frog you leave the pond, but you have to return to complete the cycle. The game completely changes (see, the "change" thing again) when you are a frog and now it's about jumping back in the water and catching bugs.

I drew some tadpoles and fish and plants and stuff, and found some great music from Incompetech and sound effects on Freesound.org. I can't say enough good things about Kevin MacLeod of Incompetech. The library of extremely high-quality music that he offers royalty-free is just awesome. And his graph paper is pretty cool too.


But what about the code? What did I do? How did I throw something together so fast?

For starters, I didn't use any game development systems. I know some that are very popular and allow you to publish your game as HTML5 and for iOS and Android and a whole bunch of other stuff, but I don't know how to use them. I admit it. I'm not a professional game developer. It's more of a hobby. I am a professional software engineer, and I do know how to write decent JavaScript so I just went with what I knew. I also kept it simple. When the game is simple, the code is simple, and doesn't require a lot of overhead from tools. Here's a quick overview of the basics. You can probably use the outline to make some simple HTML5 games yourself.

There are two absolutely essential packages for making HTML5 games: jQuery and SoundManager2. Well, jQuery might be essential for any type of web development these days. But games aren't much fun without sound and SoundManager2 is the library when you want to do sound from JavaScript. Stop what you're doing, download it, learn it, and love it. Right now.

That being said, making a simple game on an HTML5 canvas element is probably easier than you thought. To begin, create a web page with a canvas element.

<!DOCTYPE html>
<html>
  <head>
    <title>Game</title>
    <style>
     canvas { border: 1px solid #000000; } 
    </style>
  </head>
  <body>

    <div>
      <canvas id="game" width="800" height="450"></canvas>
    </div>
    
  </body>
</html>

You're halfway there! Okay, maybe not halfway. The next thing to do is set up some JavaScript to start working with the canvas. The basic outline of any game is to make a loop that updates the objects on the screen and draws them. In modern browsers, we have a nice utility for that - the window.requestAnimationFrame() function. This method has been added specifically for the purposes of doing animations on web pages, and it is the most efficient way of doing so. The following is a bare-bones version of a game. Note that you have to explicitly call requestAnimationFrame again after each call to your run function (or it will only run once!).

(function(window) {
 
 function Game() {
  
  this.lastTime = null;
  this.paused = false;

  this.canvas = null;
  this.ctx = null;
  
  this.setup = function(options) {
   
   this.canvas = options.canvas;
   this.ctx = this.canvas.getContext("2d");

   var me = this;
   window.requestAnimationFrame(function(e) { me.run(e); });

  };
  
  this.run = function(timestamp) {

   var elapsed;
   if (this.lastTime === null) this.lastTime = timestamp;
   elapsed = timestamp - this.lastTime;
   this.lastTime = timestamp;

   this.gameLoop(elapsed)
   
   var me = this;
   window.requestAnimationFrame(function(e) { me.run(e); });
   
  };
  
  this.gameLoop = function(elapsed) {
   
   // update all your objects here
   
   this.gameDraw();

  };
  
  this.gameDraw = function() {

   // clear your canvas   
   this.ctx.clearRect(0,0,this.canvas.width,this.canvas.height);
   
   this.ctx.save();

   // draw all your objects here   
   
   this.ctx.restore();
   
  };

  
 } // Game()
 
 game = new Game();
 
 window.game = game;
 
}(window));

After saving this code (I saved it as "main.js") you can fire it up by adding the following scripts into your HTML file with the canvas element:

<script type="text/javascript" src="js/jquery-1.10.2.min.js"></script>
<script type="text/javascript" src="js/main.js"></script>
<script type="text/javascript">

    $(function() {

        game.setup({
         canvas: document.getElementById("game")
        });
        
    });
      
</script>

Now as soon as your page is loaded, your setup function is called. It is passed the canvas element and starts looping and drawing. To complete your game, just add some objects to be moved around and drawn!

If you'd like to see my completed code, it's here on GitHub: cwleonard / game-off-2013.

If you'd like to play the completed game, you can find it here: The Frog Lifecycle Game.

Leave a comment below with your scores! A really good time for becoming a frog is about 135 seconds.

Friday, August 23, 2013

We're Kickstartering

Great news! My wife's company, Syslark, Inc., has decided to use Kickstarter to raise the funds to take my FFZ game from amusing prototype to finished game on the OUYA!

She's participating in OUYA's Free the Games Fund, which provides matching funds for games that will be OUYA exclusives for 6 months after launch.



So everyone please have a look and back her if you like what you see.


Monday, August 5, 2013

The Science of Fun

Let's Do Science!

Well, let's at least consider why some games are fun and you want to play them more and more...while other games don't hold your attention for 3 minutes.

Two games that I really think get it right are Minecraft and The Amazing Frog. On the surface, these games have absolutely nothing in common. Punching trees to collect wood in one and throwing a rag-doll frog off the top of a parking garage in the other. So what makes them fun?

First, both are immediately playable. There's no complex story you have to sit through. In fact, I don't know that there's a story at all. There doesn't need to be. And there's no tutorial. You just start doing stuff. You pick it up as you go. You have nothing to lose, except your currently mundane existence in which you are not playing these games.

Second, both create environments of open-ended creativity. Obviously in Minecraft you can build yourself houses or roller-coasters or chicken farms or whatever you want. You can play with friends - one of you digging in holes while the other rigs traps for zombies. Or not. You can really do whatever you want. There are no rules. But The Amazing Frog is not all that different. Okay, so it's more about blowing up cars in Swindon than it is about building things, but there aren't really a lot of rules. You get points for falling, but you don't have to care about that. My kids have also turned the game into hide-and-seek, soccer (well maybe football since it's British), and capture the flag (or crown in this case). Sometimes they just bounce in the bounce house. For like 37 minutes! And there's always something unknown on the horizon. None of us have been able to get in that cargo container hanging from the blimp yet, but it fills our imaginations with ways to maybe get there - all of which end with us blowing something up instead. What's up with that crown anyway?

Third, and this one is maybe odd, but neither game is finished. Huh? Really. Minecraft updates all the time. Do you remember how exciting it was to get carrots on fishing poles so you could finally steer pigs? You'd had pigs to ride for months, and that was fun, but now you can steer them! It's like you just got a whole new game! My daughter gets really excited when Minecraft PE updates on her iPod. New stuff! Even if she's already had that stuff in the PC version, it's still exciting. It's like when you were a kid and you loved checking the mailbox every day...some days you would get something and it would be great! Other days you would get nothing but it just made you more excited about the possibility of getting something tomorrow. These "unfinished" games give you the same feelings. Maybe "unfinished" isn't the right word. Maybe the games are complete in their state of constant change. I can't wait to see the next iteration of Amazing Frog...and if you follow Fayju on Twitter you get to see little hints of things to come that make you even more anxious. Will we get a few more stages? New outfits for the frogs? New things to grab and throw? Or whole new game mechanics? No matter what, I'm sure it will be great - and it keeps players coming back for more.

So now I will summarize.

  1. Stories. Keep them short.
  2. Rules. Keep them few.
  3. Updates. Keep them coming.
And that is how you make a great game. In theory, anyway. If we're going to be scientific about it we have to test this theory by making games according to these rules and see if they become awesome.



Thursday, July 4, 2013

Facebookified and Zyngaficated

I've decided that to truly succeed as a game developer I have to change the way I design things. I have to design for revenue. I have to put revenue before gameplay. Here's my plan...

  1. Limit how long people can play. I know this sound crazy, but just wait. I'll limit my players to 20 minutes of play and then make them wait an hour to play for another 20 minutes - UNLESS they give me another dollar. Then I'll let them play as much as they want, but only for 24 hours. Tomorrow they'll have to give me another dollar.
  2. Start out easy, then make it impossible. Will people quit playing when it gets too difficult? No, they'll already have too much of an emotional investment in the game to walk away. They'll just have to buy a power-up that will enable them to continue. It will cost another $1. Then after that really hard level it will get easier for a little while, you know, to get them back into it before I throw in another impossible level and charge another dollar.
  3. Make you bring friends if you want to keep playing. At first you can play by yourself and multiplayer will be optional. But after a certain point, you have to play with other people if you want to continue. Something in the level won't be possible without a second player. And that second player has to buy the game too! She might not want to at first, but you'll keep twisting her arm until she does. And she will because you're friends. Then BAM! I start getting dollars here and there from her too. Brilliant!
Oh, wait. I forgot. I just want to make things that are fun. But this plan sounds strangely familiar. It sounds like the outline of every Zynga and/or Facebook game ever made.

Is this all the world has to offer? I get to choose between another pointless first-person shooter that costs me $70 up front, or an addictive puzzler that is designed to take my money one dollar at a time?

This is why I'm developing for OUYA.

Saturday, June 15, 2013

Hack It, then Refactor It

I've spent the last several days refactoring my OUYA game. It seems strange to spend so many hours working on the code and the great accomplishment is that it still works. I just felt the need to explain why I do this and how it works for me.

I believe that I am a very results-oriented person. Not to say that the ends always justify the means, but I recognized many years ago that getting a product working is much more important that making sure that all the T's are dotted and the I's are crossed. I've also been known to get forgiveness instead of permission on many occasions.

You can be writing the most elegant, perfect, optimized code in all the world but if no one ever sees or uses the product then what have you really accomplished? This is true for everything, not just software products. Once (1998) I decided to replace the stock 3.8L V6 engine in my 1989 Mercury Cougar with a 5.8L small-block V8. I had 3 months until I needed the car to run again so I could go back to college. As far as we could tell, this feat of automotive engineering had never before been accomplished. The engine did not physically fit in the car. But I wasn't building it to put on display, so we hacked and bent and cut and torched and pounded with a hammer. It wasn't pretty, but it ran. I drove it back to college and almost every day while I was there. I also spent the next 3 years fine-tuning it. Parts were replaced with new ones as they were fabricated. The more powerful engine totally blew out the rest of the drive-train in time, but that got fixed as well. I could have spent those 3 years making it perfect before it ever left the garage, but it was much better to be driving it all that time.

Basically, I learned that success means shipping a product. So how does this relate to hacking and refactoring software? It's pretty much the same thing.

I wanted to get stuff on the screen of the OUYA and get things moving around with the controller as quickly as I could. I needed to see it working, even if the code was less than perfect. I needed to see those results so that I could continue. Letting it "sit in the garage" for too long would just discourage me and I would lose interest.

However, I know better than to leave the code that way for too long. If I am going to build a complete game, my code has to be structured such that it can be built on. It can't be brittle. It has to use the proper design patterns. So every few weeks, I stop adding new features and refactor. I clean up the ugly code and make sure I have a solid foundation before building more. But in the mean time I also have something that my kids and I can play (You don't have to release your product to everyone to count it as a release. My Cougar only had one user, after all).

It's just what works for me. And it works for other people as well. Check out the works of Kent Beck and Martin Fowler. They've written lots of stuff on design patterns and refactoring code.

Also, read this paper.

And as for my Cougar, I no longer drive it but it taught me a lot about successful engineering projects.