Blog | object.io

whiskey_disk makes it very easy to quickly deploy a new version of your site to one or more servers. It’s very efficient, as files are copied directly from your git repository to the server. This means you can very quickly make large deployments even when you are on a slow internet connection. You do need a secure way for your server to access the git repository while the deploy is going on, read on to see how this is easily done.

For this example, I’m going to deploy a small static site from a private Github repository to a server via ssh using whiskey_disk. There are a few steps in getting this all setup, but once you have the ssh infrastructure playing nicely, a complete deployment of a small static site takes on the order of 2-5 seconds. Stay with me, and I’ll show how all the pieces fit together.

To begin with, I’ll assume you have a site or application already pushed to a private Github reposity that you now want to do a deploy of. You also have ssh access to the server you are deploying to for this to work. For this example, I’m deploying the master branch of the following private git repository:

git@github.com:rud/whiskey_disk-demo.git

Adding the configuration file

Make sure you have your appropriate code pushed to Github, then we’ll add the whiskey_disk config file config/deploy.yml to the project:

demo:
  domain:     "deploy_user@example.com"
  deploy_to:  "/home/deploy_user/domains/useful_site.com"
  repository: "git@github.com:rud/whiskey_disk-demo.git"

This is a simple YAML file (indentation matters), you will need to adjust the domain and paths as appropriate. I use the name demo for the environment, other common examples would be staging, production, etc. A lot more options are available, see the official whiskey_disk readme, but this all we need to get started.

Running whiskey_disk setup

whiskey_disk has two commands: a setup command, for doing the initial repository clone, and the deploy command used for all subsequent deploys.

With the config/deploy.yml file in place inside the project, we’re going to just try and run the setup command from the local computer, seeing what happens. For now, this errors out:

$ wd setup -t demo
Initialized empty Git repository in /home/deploy_user/domains/useful_site.com/.git/
Host key verification failed.
fatal: The remote end hung up unexpectedly
fetch-pack from 'git@github.com:rud/whiskey_disk-demo.git' failed.
error: pathspec 'origin/master' did not match any file(s) known to git.
Did you forget to 'git add'?

Okay, we’re not quite ready to do deployments just yet.

Resolving all ssh issues

Github hangs up on us - what’s happening? Testing from my local machine as described in the github ssh guide:

$ ssh -T git@github.com
Hi rud! You've successfully authenticated, but GitHub does not provide shell access.

That’s the kind of success we’re looking for on the server as well. Ssh to the server as the deploy user, and try the same command there:

$ ssh -T git@github.com
The authenticity of host 'github.com (207.97.227.239)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,207.97.227.239' (RSA) to the list of known hosts.
Permission denied (publickey).

Try the command again, now that the host key is trusted:

$ ssh -T git@github.com
Permission denied (publickey).

Notice how we get a new error later in the connection process, a Permission denied (publickey) instead of Host key verification failed.

For the sake of completeness, this is what this situation looks like from whiskey_disk:

$ wd setup -t demo
Initialized empty Git repository in /home/deploy_user/domains/useful_site.com/.git/
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
fetch-pack from 'git@github.com:rud/whiskey_disk-demo.git' failed.
error: pathspec 'origin/master' did not match any file(s) known to git.

ssh-agent to the rescue

The root cause for this difference is Github knows my public key and has access to the corresponding private key, but this key is not available to my server, so it cannot authenticate against Github. The solution is fortunately simple. Do this on your local machine:

$ ssh-add

This prompts you for the password for your private key, then keeps the private key unlocked in memory for automated access. You need to do this after each time you restart before you can do new deployments, as the unlocked key is only stored in memory. We also need to enable ssh-agent support for the connection to the server - as ssh-agent gives a server the possibility to use our private key without further prompting, you need to trust the administrator of servers you deploy to. With that caveat out of the way, open your ~/.ssh/config file, and add a config block like this:

Host example.com
  ForwardAgent yes  # This enables ssh-agent functionality, making whiskey_disk happy

Adjust the hostname to taste, obviously. You can also add port-number here, if your admin moved ssh access from the default port. While you have that file open, also check out these extremely handy ssh productivity tips for a lot of extra ssh goodness.

Close the previous connection to your server, open a new one (so ssh-agent can do its magic), then re-run the test:

$ ssh -T git@github.com
Hi rud! You've successfully authenticated, but GitHub does not provide shell access.

We have ssh-win!

Deploying

Time to re-try the deployment setup from the local machine:

$ wd setup -t demo
Results:
deploy_user@example.com => succeeded.
Total: 1 deployment, 1 success, 0 failures.

This means the repository was successfully cloned into place. We only have to do this once, from now on deployments are just a matter of:

$ wd deploy -t demo
Results:
deploy_user@example.com => succeeded.
Total: 1 deployment, 1 success, 0 failures.

I’m seeing a total time taken for a deployment in the area of 2-5 seconds. Remember to push to Github before running your deployment, as it is from Github your server is pulling versions.

This method for deployment is extremely awesome, and the amount of awesome grows with the size of your deployments, as server-server connections are most likely a lot faster than your own internet connection.

Next steps

There is a lot more to whiskey_disk that I’ve shown here. In particular I’d like to highlight the “Configuration Repository” section of the official whiskey_disk readme. Editing per-application configuration files on individual servers is effectively a thing of the past.

Years ago I read Streamed Lines: Branching Patterns for Parallel Software Development, and many of the thoughts have stayed me. Think of it as a catalogue of Design Patterns for version control workflows. There are so many ways to efficiently use branches to both collaborate and isolate work. Lively discussions can be had on how to best organize a repository, and having a great overview of the options makes for more informed discussions.

I have previously described how I’ve been using small hack && ship commands to simplify and streamline my personal git workflow. Recently I’ve switched to using the Git Flow setup as my branching and release workflow.

Setting up git-flow

Installing the git-flow toolset on OS X is trivial using the homebrew installer:

$ brew install git-flow

The git-flow project also has instructions for installation on Linux and Windows.

Installing git-flow adds a few helpful commands to the git environment for creating and managing branches for features and releases. A fresh git repository is born with a master branch. In the default git-flow setup, this is where the current production release lives. Furthermore, a branch called develop is created, this is where development takes place. Note that git-flow is just a series of shortcuts to having a development branch and a production branch with sensible ways of shuttling changes back and forth. After installing the git-flow package, configure your local repository for git-flow use with:

$ git flow init

You can most likely accept the defaults by pressing enter at each question - this also makes it easier for others to initialize git-flow in their working copy, as all are using the same defaults. The only change from running the init command is two [gitflow] sections in your .git/config file looking like this:

[gitflow "branch"]
  master = master
  develop = develop
[gitflow "prefix"]
  feature = feature/
  release = release/
  hotfix = hotfix/
  support = support/
  versiontag =

It’s no more magic than that. Consult the built-in help with:

$ git flow

If you’ve got the bash-completion package installed, you can use tab-completion for git-flow commands and arguments. Highly recommended.

Working on a feature

Start work on a feature - say feature 77 from your issue-tracker:

$ git flow feature start 77-speedup-yak-shaving

This creates a new regular local git branch called feature/77-speedup-yak-shaving based on the current develop and places you on it. Want to share your work in progress with others? Use:

$ git flow feature publish 77-speedup-yak-shaving

This pushes the branch, and sets up your local branch to track the remote branch in one easy step. The regular git push and git pull --rebase work as you would expect here. There is nothing special about branches created by git-flow, they are just conveniently named and managed.

All done with a feature? Rebase your feature on the current develop branch, then merge it in:

$ git flow feature rebase
$ git flow feature finish 77-speedup-yak-shaving

With these two steps, you end up on develop with your feature merged in. Run your test-suite, and push to a central repository as appropriate.

Releasing to production version and hotfixes

Production releases are handled quite nicely in git-flow:

$ git flow release start 2011_year_of_the_yak

This creates a new branch called release/2011_year_of_the_yak, based on the current develop branch. Here you can fix any HISTORY or VERSION file, commit, and the finish up the release with:

$ git flow release finish 2011_year_of_the_yak

By finishing the release, you now get a tag called 2011_year_of_the_yak, and the temporary release-branch is removed. You end up back at the master branch.

A hotfix is simply a feature branch which is based on the latest production release, and which is automatically merged to both develop and master. Extremely handy.

Rebase considered harmful?

Some developers prefer to only ever merge, but I’m a big fan of rebasing local work before publishing. I’ll leave you with this:

If you treat rebasing as a way to clean up and work with commits before you push them, and if you only rebase commits that have never been available publicly, then you’ll be fine. If you rebase commits that have already been pushed publicly, and people may have based work on those commits, then you may be in for some frustrating trouble.

From Pro Git, chapter 3

Deploy a private Github repository with whiskey_disk