More Automated Deployment

I'm still working on my automatic deployment system for the webcomic code. One thing that was left out of my first version was special handling for certain types of file updates.

For example, if I add a new package to package.json I'll need to run npm install after pulling the changes if I want the application to work. Fortunately, GitHub includes lists of files added, removed, and modified as part of the data POSTed to my listener.

If you want to see exactly what GitHub sends you from these Webhooks, just go to the management page for one of them. At the bottom, there's a list of all the POSTs they've sent (or attempted to send, in case some of them have failed). Click on the ids to expand the details and you'll see the JSON. I just copied and pasted it into JSON Editor Online so that I could easily examine it.

GitHub's Webhook transmission history - check it out!

If you look at them, you'll notice that the "head_commit" field has three fields named "added", "removed", and "modified." As you might expect, these fields are arrays of file names that were added, removed, or modified. I made a simple function to search through these arrays and look for package.json. If I find it, I know that I'll have to run the npm install command after the pull is complete.

function checkMods(listOfLists) {

    var ret = {
        runNpm: false,
        restartApp: false
    };

    for (var j = 0; j < listOfLists.length; j++) {
        var fileList = listOfLists[j];
        for (var i = 0; i < fileList.length; i++) {
            if (fileList[i] === "package.json") {
                ret.runNpm = true;
            }
        }
    }

    return ret;

}

My function returns an object that informs the caller if npm should be run and also includes a flag for telling the caller if the app should be restarted as well. I'm thinking ahead but haven't implemented that check quite yet.

I modified the "push" listener from earlier in the week to look like this now:

github.on("push", function(repo, ref, data) {

    var branchName = ref.substring(ref.lastIndexOf("/")+1);
    if (branchName === "master") {

        var repoName = data.repository.name;
        if (repoName === "comic") {

            var checks = checkMods([data.head_commit.added,
                                    data.head_commit.removed,
                                    data.head_commit.modified]);

            var toRun = "/home/comic/update_comic.sh";
            if (checks.runNpm) {
                toRun += " npm";
            }

            child = exec(toRun, function(error, stdout, stderr) {

                if (error !== null) {

                    var e = new Error("exec error: " + error);
                    console.log(e);

                } else {

                    console.log("pulled update for " + repoName);
                    if (checks.runNpm) {
                        console.log("...and installed new packages");
                    }

                }

            });

        }

    }

});

The only real change is the call to checkMods on line 9 that passes in an array of arrays, and the subsequent conditional addition of the "npm" flag on the Bash script that gets called. Obviously, that flag indicates to the script that npm install should be executed after git pull origin master.

Now I just need to get to work on that conditional restart part. In theory, I just have to look for changes to server-side .js files (the .js files not under public) and then flag the app for a restart. The biggest issue for me is my lack of Bash scripting skills. But I'll figure it out.

Soon, I'll have even more autonomous electronic robot mackerels in my sea.

If you're confused by that last statement, see today's comic:

Amphibian.com comic for 1 May 2015

Automating Code Deployment with GitHub Webhooks and Node

Do you push a "pull" button?

I've been pushing updates to my webcomic's code repo on GitHub for over a year now. My workflow has gone something like this:

1. Make code changes.
2. Test locally.
3. Push changes to GitHub.
4. Log on to production server.
5. Pull changes from GitHub.

I'm trying to automate steps 4 and 5. I'd like to have my production server automatically update itself whenever I push code to the "main" branch of the repository.

This week I've been experimenting with using GitHub's Webhooks to automate my code deployment. Webhooks are a cool idea - set one up on your repo and every time an event occurs (push, pull request, fork, etc.) GitHub will send you some data to a URL of your specification which describes the event. All you need is something to process the data!

I set myself up a simple Node application to process the data using the githubhook package. It was very easy to use.

var githubhook = require("githubhook");
var exec = require("child_process").exec;

var github = githubhook({
    port: 8082,
    secret: "your-secret-code"
});

// listen to push on github on branch master
github.on("push", function(repo, ref, data) {

    var branchName = ref.substring(ref.lastIndexOf("/")+1);
    if (branchName === "master") {

        var repoName = data.repository.name;
        if (repoName === "comic") {

            child = exec("/home/comic/update_comic.sh", function(error, stdout, stderr) {

                if (error !== null) {

                    var e = new Error("exec error: " + error);
                    console.log(e);

                } else {

                    console.log("pulled update for " + repoName);

                }

            });

        }

    }

});

// listen to github push
github.listen();

My application will listen on port 8082 for GitHub events, which will be HTTP POSTed in JSON format (when you select that option later). When setting up, the value for secret (line 6) is important. You'll want that to match the value you'll set up in GitHub. Once you create the object, specify callback functions for the events that interest you and start listening.

I am just looking for the "push" event. When that event happens, I check to make sure the push was to the master branch and that the repo was my comic (I might use this for other repos later). If both of those are true, I exec a script that will pull the updates and restart the app if necessary.

Once you have your app running, you can set up your GitHub repo with the Webhook. Just go to your repository page, and click on the Settings link. Then click on "Webhooks & Services" and then on the "Add webhook" button.

Now specify the URL where you are listening with the app. My URL happens to be http://caseyleonard.com:8082/github/callback. The port is obviously the same one I specified above, but the path of /github/callback is the default used by githubhook (you can change it if you want - see the docs). Set the Content Type to be "application/json" and type in the same value for Secret that you used in the app. If these don't match, GitHub will receive a 403 response code when it sends you data and your listener will not be called. You can also select which events of which you want to be notified. It defaults to just "push" which is fine for my use.

It's simple and it works for me, at least so far. The less time I spend updating the code for the comic, the more time I can spend making the comics themselves. Today is the last of the Canterbury Tales parody, a set which took me a very long time to create!

Amphibian.com comic for 29 April 2015