Multi-node topologies using Vagrant and chef-zero

I’ve been bootstrapping a lot of multi-node topologies recently. Here’s some things I learned about making it easier with Vagrant and chef-zero. I also wrote a Chef knife plugin (knife-topo) – but more of that in a later post.

Multiple VM Vagrant files

The basics of a multi-VM vagrant file as described in the Vagrant documentation is that you have a “config.vm.define” statement for each machine, and setup the machine-specific configuration within its block. This looks something like:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu64"
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
  config.vm.synced_folder "/ypo", "/ypo"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  # setup appserver
  config.vm.define "appserver" do |appserver_config|
    appserver_config.vm.hostname = "appserver"
    # other VM configuration for appserver goes here
  end

  # setup dbserver
  config.vm.define "dbserver" do |dbserver_config|
    dbserver_config.vm.hostname = "dbserver"
    # other VM configuration for dbserver goes here
  end
end

Lines 2-3 tell Vagrant what to put on the machines, and where to get it. Lines 4-5 are optional: line 4 sets up a directory on the host machine which will be accessible to each guest virtual machine; line 5 disables the default share.

Lines 7-10 and 13-15 each define a virtual machine (appserver and dbserver).  The two config variables (appserver_config and dbserver_config) are like the overall ‘config’ variable, but scoped to a specific machine.

This is fine when you don’t have much configuration information for each node, but it gets hard to read, laborious and error-prone to maintain when you start having lots of configuration statements or lots of nodes.

Separating out the configuration of the virtual machines

I got this approach from a post by Joshua Timberman in the days before I learned Ruby, and it was a great help in cleaning up my Vagrantfile.  Separating out the machine (node) configuration makes it so much easier to understand and update the Vagrant file.

First, we define a structure (Ruby hash) that describes how we want the nodes to be configured:

nodes = {
  :dbserver => {
    :hostname => "dbserver",
    :ipaddress => "10.0.1.2",
    :run_list => [ "role[base-ubuntu]", "recipe[ypo::db]" ]
  },
  :appserver => {
    :hostname => "appserver",
    :ipaddress => "10.0.1.3",
    :run_list => [ "role[base-ubuntu]", "recipe[ypo::appserver]"],
      :forwardport => {
        :guest => 3001,
        :host => 3031
    }
  }
}

The above defines two nodes, dbserver and appserver.  Lines 3-5 and 8-10 set up their hostnames, IP addresses and the run lists to use with Chef provisioning. Lines 11-13 set up port forwarding, so that what the application server listens for on port 3001 will be accessible through port 3031 on the host machine.

Second, we use the nodes hash to configure the VMs, with code something like this:

Vagrant.configure("2") do |config|
  nodes.each do |node, options|
    config.vm.define node do |node_config|
      node_config.vm.network :private_network, ip: options[:ipaddress]
      if options.has_key?(:forwardport)
        node_config.vm.network :forwarded_port, guest: options[:forwardport][:guest], host: options[:forwardport][:host]
      end
    end
    node_config.vm.hostname = options[:hostname]
    #
    # Chef provisioning options go here (see below)
    #
  end
end

Line 2 loops through the nodes hash, defining the configuration of each machine. Line 4 sets up the machine on a private network with a fixed IP address as specified in the nodes hash. Lines 5-7 setup port forwarding, if configured in the nodes hash. Line 9 sets up the hostname from the nodes hash.

Multiple nodes with a single chef-zero

chef-zero is an in-memory Chef server – a replacement for chef-solo. It’s a great way to test recipes, because it is just like a ‘real’ chef server. By default, it listens only locally, on http://127.0.0.1:8889. However, you can run it so that other nodes can access it – for example, all of the VMs in the Vagrantfile.  Doing this lets you use recipes requiring search of other nodes’ attributes, e.g. for dynamic configuration of connections between nodes. When you’re done testing, you just stop chef-zero – no cleanup of the server required.

You can install chef-zero as a gem:

gem install chef-zero


The machines defined above are all on the 10.0.1.x private network. So all we need to do is start chef-zero on that network, using:

chef-zero -H 10.0.1.1

and configure the VMs with a chef server URL of “10.0.1.1:8889”. Because the Vagrantfile is reading configuration from the knife.rb file, all we need to do is set variables in knife.rb – something like:

node_name                "workstation"
client_key               "#{current_dir}/dummy.pem"
validation_client_name   "validator"
validation_key           "#{current_dir}/dummy.pem"
chef_server_url          "http://10.0.1.1:8889"

The dummy.pem can be any validly formatted .pem file.

Using Chef provisioning with Vagrant

Running chef-client using Vagrant provisioning

Sometimes it is very useful to have Vagrant provision your machines using chef-client. The Vagrant documentation provides you with the basics for doing this, but not a lot else. The first useful tip (again from Joshua Timberman) is to use your knife.rb configuration file rather than hard-coding the server URL and authentication information. To do this, add the following at the top of the Vagrantfile:

require 'chef'
require 'chef/config'
require 'chef/knife'
current_dir = File.dirname(__FILE__)
Chef::Config.from_file(File.expand_path('../.chef/knife.rb'))

These five lines read in the configuration from your knife.rb file (change ‘../chef/knife.rb” to be the path to your knife.rb) and make it available to be used as below, which shows code to that should be inside the “nodes.each” block described previously:

node_config.vm.provision :chef_client do |chef|
  chef.chef_server_url = Chef::Config[:chef_server_url]
  chef.validation_key_path = Chef::Config[:validation_key]
  chef.validation_client_name = Chef::Config[:validation_client_name] 
  chef.node_name = options[:hostname]
  chef.run_list = options[:run_list]
  chef.provisioning_path = "/etc/chef"
  chef.log_level = :info
end

Lines 2-4 setup the connection to the Chef server. Lines 5-6 setup the node name and run list from the options hash we defined earlier. Setting the provisioning_path to “/etc/chef” puts the client.rb in the place that chef-client expects it, and setting log_level to “:info” provides a decent level of output (change to “:debug” if you have problems with the provisioning).

Cleaning up the Chef Server on vagrant destroy

When you destroy a Vagrant VM that you have provisioned using a Chef Server, you need to delete the node and client on the Chef server before you reprovision it in Vagrant. The following lines added into the node_config.vm.provision block above should do this for you:

chef.delete_node = true
chef.delete_client = true

However, there is an issue with this – it does not work for me, so I always have to use knife to manually clean up:

knife node delete appserver
knife client delete appserver


Controlling the version of chef-client

Initially I was running into issues because the chef client used by the Vagrant chef provisioner was back-level. This post on StackOverflow solved the problem for me. Install the chef-omnibus Vagrant plugin:

vagrant plugin install vagrant-omnibus

and specify the version of chef-client you want to use in the node configuration part of the Vagrant file:

node_config.omnibus.chef_version = "11.12.8"

or:

node_config.omnibus.chef_version = :latest

to install the latest version.