Tidbits from creating a Chef Delivery application pipeline

This post contains an assortment of things I learned whilst creating a Chef Delivery pipeline for a nodejs application, presented in no particular order. It’s really the post that never grew up. I’m assuming that you have at least worked through the Chef Delivery tutorial before trying to read this.

First, some overall context

The examples I use in this post come from the Delivery pipeline for a nodejs application with a backend MongoDB database. Build is done with grunt, test with jasmine, deployment with Chef.

We manage a number of application-specific cookbooks in the same git repo as our application code.  These cookbooks and the application code travel down the same delivery pipeline. Other more general cookbooks that we use have their own pipelines. At some point, we might specify these as delivery dependencies for the application pipeline, but we have not yet done so.

Our git repo looks something like this:

.delivery/                 # build cookbook & delivery config
cookbooks/                 # our application-specific cookbooks             
public/                    # our client code (javascript)
server/                    # our server code (nodejs)
spec/                      # application unit and func tests (jasmine)
topologies/                # knife-topo JSON files describing deployments
Gruntfile                  # our grunt build scripts

We use delivery-truck to manage the cookbooks through the pipeline. In the Chef Tutorial, delivery-truck is used with a single cookbook repo: it also handles multi-cookbook repos like ours (as long as the cookbooks are in a top-level cookbooks directory).

We wrote our own build cookbook logic (not suprisingly) to manage our application code through the pipeline. In general, that logic consists of (1) what is needed to setup the build tools and (2) a thin wrapper around existing build and test scripts.

An example of one of our phase recipes is the following, for publish:

include_recipe 'build-cookbook::_run_config'
include_recipe 'delivery-truck::publish'
include_recipe 'topology-truck::publish'
include_recipe 'build-cookbook::_publish_app'

The _run_config recipe does some standard setup in each phase.

The delivery_truck::publish recipe looks for any changed cookbooks, and if there are it unloads them to the Chef Server.

The topology-truck cookbook is a work-in-progress that we use to provision and configure our application deployments: its topology-truck::publish recipe looks for changes to the topology JSON files, and if there are any, it uploads them to the Chef Server (as data bag items) using the knife-topo plugin.

The _publish_app recipe packages up the application code, and publishes the package to an internal repository so that it can be used in the later deploy phase.

In this pipeline, we do not literally push to production in the ‘Delivered’ stage. Instead, the output is a build of the application that is ready to be deployed in production. For us, the ‘deploy’ actions in this stage are more a final ‘publish’.

Default recipe – install build tools for use in multiple stages/phases

In each stage, Chef Delivery runs the build cookbook default recipe with superuser privileges, and then runs the specific phase recipe as a specific user (dbuild). The dbuild user has write access to the filesystem in the project workspace, but cannot be used to install globally.

Quite often, there will be some build tools that you either have to install with superuser access, or that it makes sense to install once for use across multiple phases (in which case, you probably do not want to share build nodes across projects). You will need to use the default recipe to setup these tools.

Here’s an example from our default recipe, where we install some prerequisite packages, nodejs and the grunt client:

# include apt as it's important we get up to date git
include_recipe 'apt'

%w(g++ libkrb5-dev make git libfontconfig).each do |pkg|
  package pkg

include_recipe 'nodejs::default'

# enable build to run grunt by installing client globally
nodejs_npm 'grunt-cli' do
  name 'grunt-cli'
  version node['automateinsights']['grunt-cli']['version']

If you used the above, you would discover that phases can take some considerable time to complete, even if there is no work to be done. What’s going on?

It turns out that the file cache path is being set to a path in the unique workspace for that particular project, stage and phase (e.g. /var/opt/delivery/workspace/ In each stage and phase the default recipe (via the ark resource) downloads the nodejs package to that unique directory. That can be a lot of slow downloads, even if the correct version of nodejs is already installed.

We prevent this by changing the cache path, for the default recipe only:

# Force cache path to a global path so default recipe downloads
# e.g. nodejs don't get put in each local cache
Chef::Config[:file_cache_path] = '/var/chef/cache'

Phase recipes – beware community cookbooks needing superuser

Most community cookbooks assume they will be run with superuser privileges. You may encounter issues if you try to use them in phase recipes rather than in the default recipe.

For example, we usually use the nodejs_npm resource from the nodejs cookbook to install npm packages for our application. It should be fine to do this in the phase recipe, because we’re installing the npm packages locally. Unfortunately, the nodejs_npm resource always checks whether nodejs is installed by running the install recipe. This action fails without superuser privileges, even if nodejs is already installed.

Basically, you will often end up using the execute resource in the phase recipes e.g.:

repo_path = node['delivery']['workspace']['repo']
build_user = node['delivery_builder']['build_user']

# install test dependencies cannot use nodejs_npm
# because that will try to install nodejs and fail for permissions
execute 'install dependencies' do
  command 'npm install'
  cwd repo_path
  user build_user

A little bit of sugar

The delivery-sugar cookbook has some useful helpers for your recipes. Include it in your build cookbook’s metadata.rb to use it.

change.stage is useful if you need conditional logic depending on the stage (e.g. your logic for provision or deploy may vary in Acceptance, Union, Rehearsal, Delivered).

change.changed_cookbooks and change.changed_files are useful if you need logic dependent on what has changed in the commit.

Probably the most useful sugar is with_server_config, illustrated in the next section.

Accessing the Chef Server from phase recipes

The build cookbook recipes are run using chef-client local mode. If you try to do a  node search or access a data bag in the build cookbook recipes, it will not work as you intend. Luckily delivery-sugar can help here, too. The with_server_config command will let you do what you want. It temporarily changes the Chef::Config to point to the Chef Server, and then switches it back to the local context.

Here’s a recipe where we use Chef vault to setup AWS credentials on a build node. The highlighted code executes at compile time in the context of the Chef Server to retrieve the credentials. The rest of the recipe executes in the local context.

# setup credentials to push package to S3
include_recipe 'chef-vault'

root_path = node['delivery']['workspace_path']
aws_dir = File.join(root_path, '.aws')

creds = {}
with_server_config do
  creds = chef_vault_item('test', 'aws_creds')
build_user = node['delivery_builder']['build_user']

directory aws_dir do
  owner build_user
  mode '0700'

template File.join(aws_dir, 'config') do
  source 'awsconfig.erb'
  owner build_user
  mode '0400'
    user: build_user,
    region: node['myapp']['download_bucket_region']

template File.join(aws_dir, 'credentials') do
  source 'awscredentials.erb'
  owner build_user
  sensitive true
  mode '0400'
    user: build_user,
    access_id: creds['aws_access_key_id'],
    secret_key: creds['aws_secret_access_key']

If you want to do something similar to the above using Chef Vault, remember to add the build nodes to the search query for the vault.

Set your own log level

By default, Chef Delivery displays the WARN log level in the output from each phase. You may well want to use a different level.  In your build cookbook, create a recipe e.g. recipes/set_log_level.rb:

# Set config for all runs

Set the default log level that you want in the build cookbook in an attribute file, e.g: attributes/default.rb:

default['delivery']['config']['myapp']['log_level'] = :info

Include the set_log_level recipe in all of the phase recipes (unit.rb, lint.rb, syntax.rb, etc) and in the default recipe:

include_recipe 'build-cookbook::set_log_level'

You can then override the default value for log level in the .delivery/config.json file:

  "myapp" : {
    "log_level": "debug"

Node attributes available to phase recipes

The following is an example of the node attributes passed into a Delivery build run and thus made available to build cookbook recipes. node['delivery_builder']['build_user'] and node['delivery']['workspace']['repo'] are particularly useful, as is the ability to pass in attributes via the delivery config.json and have them appear under node['delivery']['config'], as shown in the previous section.

                ... as specified in config.json...


Chef Delivery on a laptop

This post describes how you can run Chef Delivery on a laptop, using Vagrant. My main intent is to give you a way to work through Chef’s Delivery tutorial if you do not have access to AWS – or if you are really lost using a Windows workstation. Use the AWS CloudFormation template in the Tutorial if you can – the approach in this post is more error-prone. You have been warned.

I’m going to assume you are reasonably familiar with Linux, Vagrant and using the knife command – and that you already have Vagrant and Virtualbox installed and working.

We’re going to spin up multiple (at least 4) virtual machines. I’m using an 8-core Ubuntu laptop with 16GB memory. If you are running on something much smaller, good luck.

Plan of Attack

The first part (the bulk) of this post, Setting up Chef Delivery, replaces the Install Chef Delivery on AWS section of the tutorial. In it, we’ll use the delivery-cluster cookbook to provision a Chef Server, Delivery Server and build node as virtual machines using Vagrant.

The second part, Configuring the Workstation, sets up a workspace to use with Chef Delivery. If refers to and partly replaces the second and third sections of the Tutorial.

The final part of this post, Following the Chef Delivery Tutorial gives you some pointers on working through the remaining sections of the Chef Delivery tutorial (the ‘meat’ of the tutorial) using this setup. In particular, the first time you send the demo application through the pipeline, it won’t quite work, due to an interesting ‘chicken and egg’ problem. I’ll explain why and what to do about it when we get there.

Setting up Chef Delivery


  • Vagrant
  • Virtualbox
  • ChefDK 0.15.5 or more recent (includes Delivery CLI)
  • Git
  • build-essential (Ubuntu) or comparable development library

If you already have ChefDK installed, please check its version and upgrade if needed:

chef --version
 Chef Development Kit Version: 0.15.15
 chef-client version: 12.11.18
 delivery version: 0.0.23 (bf89a6b776b55b89a46bbd57fcaa615c143a09a0)
 berks version: 4.3.5
 kitchen version: 1.10.0

If it’s an earlier version, you may not have the Delivery CLI and you may encounter errors when using delivery-cluster.

You need the development environment library appropriate to your OS, e.g. for Ubuntu:

sudo apt-get install build-essential

(see Chef Delivery docs for other platforms)

Obtain Chef Delivery license

If you do not have a license already, you can obtain a temporary license that will let you use Chef Delivery for the tutorial.

Copy the license into the home directory on your workstation.

cp delivery.license ~

The delivery-cluster cookbook will handle putting the license onto the Delivery Server, once created.

Prepare to provision using delivery-cluster

Clone the delivery-cluster repo from Github:

git clone https://github.com/opscode-cookbooks/delivery-cluster.git ~/delivery-cluster

Run the following from within that repo to generate the provisioning settings to be used:

cd ~/delivery-cluster
rake setup:generate_env

Accept the default of test for Environment Name and Cluster ID.  Change the Driver Name to vagrant:

Global Attributes
Environment Name [test]: 
Cluster ID [test]:

Available Drivers: [ aws | ssh | vagrant ]
Driver Name [aws]: vagrant

Accept the default SSH Username and optionally change the Box Type and Box URL to use Ubuntu (which is what I am using). The default Centos selection should also work.

Driver Information [vagrant]
SSH Username [vagrant]: 
Box Type:  [opscode-centos-6.6]: opscode-ubuntu-14.04
Box URL:  [....]: https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box

Here’s the Ubuntu Box URL for easier copy/paste:


Take the defaults for all other settings. You can see all of the expected settings in the output JSON shown in the next Section.

Be aware this setup is slightly different from the Tutorial – specifically:

  • the IP addresses used for the Chef Server, Delivery Server and build node(s) are ‘33.33.33.xx’ rather than ‘10.0.0.xx’
  • the Delivery Enterprise name is ‘test’ rather than ‘delivery-demo’

The rake command will generate the environment in ~/delivery-cluster/environments/test.json. You can rerun the command or edit the file directly if you need to.

Update the environment to include FQDN

We need to make a manual update to the Delivery environment file to work around an issue  with provisioning Delivery to vagrant.  Without this workaround, various configuration files will incorrectly use an IP address of when trying to communicate with the Chef Server or Delivery Server. To avoid this, we need to specify the FQDN to be used for these servers.

Edit the file ~/delivery-cluster/environments/test.json so that the FQDN is specified:

  "name": "test",
  "description": "Delivery Cluster Environment",
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "override_attributes": {
    "delivery-cluster": {
      "accept_license": true,
      "id": "test",
      "driver": "vagrant",
      "vagrant": {
        "ssh_username": "vagrant",
        "vm_box": "opscode-ubuntu-14.04",
        "image_url": "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
        "key_file": "/home/test/.vagrant.d/insecure_private_key"
      "chef-server": {
        "fqdn": "",
        "organization": "test",
        "existing": false,
        "vm_hostname": "chef.example.com",
        "network": ":private_network, {:ip => ''}",

        "vm_memory": "2048",
        "vm_cpus": "2"
      "delivery": {
        "fqdn": "",
        "version": "latest",
        "enterprise": "test",
        "license_file": "/home/test/delivery.license",
        "vm_hostname": "delivery.example.com",
        "network": ":private_network, {:ip => ''}",
        "vm_memory": "2048",
        "vm_cpus": "2",
        "disaster_recovery": {
          "enable": false
      "builders": {
        "count": "1",
        "1": {
         "fqdn": "",
         "network": ":private_network, {:ip => ''}",
          "vm_memory": "2048",
          "vm_cpus": "2"

Note: It may not be necessary to specify the FQDN for the build node.

Provision the Servers

To start provisioning run:

export CHEF_ENV=test
rake setup:cluster

Watch it for a few minutes, to make sure there’s no early failure. You should see it create a Vagrant machine for the chef server and start to run recipes to install and configure the server.

If it’s going OK, go for coffee. Actually, go for lunch. This will take a while. There’s a lot going on… not only is it installing the Chef Server, Delivery Server and Build node, but also setting up credentials and certificates.

Sometimes a download will timeout and the provisioning run will fail part way through. If this happens, try rerunning it.

Hopefully you will come back and find a successfully completed Chef run. The last node provisioned should have been the build node.

Now we need to make sure it actually worked.  Let’s start by getting the information that will let us logon to the Servers. Run the following rake command:

rake info:delivery_creds
Username: delivery
Password: XSomeGeneratedPassword=
Chef Server URL:

Delivery Server
Created enterprise: test
Admin username: admin
Admin password: +cAnotherGeneratedPasswordg=
Builder Password: UtAndAnotherGeneratedPassword4=
Web login:

You should be able to logon to both the Chef Server and the Delivery Server using the URLs and credentials provided by the rake command. You will need to confirm a security exception with the browser, as we’re using self-signed certificates.

If you’re using Firefox and have previously installed Delivery, you may need to clear the old certificates from the browser first. Firefox will give you Error code: SEC_ERROR_REUSED_ISSUER_AND_SERIAL. Go to ‘Preferences > Advanced > Certificates > View Certificates’ in the Firefox menu and delete the entries for from the ‘Server’ and ‘Authorities’ tabs.

For a final smoke test, run the following knife command from within the delivery-cluster repo:

knife node status
build-node-test-1    available

Chef Delivery uses push-jobs, and the above command lists nodes that are visible to push-jobs. If you do not see your build node(s) when you run the command, something has  gone wrong (Chef server certificate problem, incorrect IP address, ….). Double check your environment file and try rerunning the rake command.

If you do make a significant change to the environment settings (e.g. changing the box type), I recommend destroying the virtual machines (see next section) and starting with a fresh clone of delivery-cluster, to remove cached provisioning information.

Managing the virtual machines

The delivery-cluster cookbook creates Vagrant specifications for the virtual machines in ~/delivery-cluster/.chef/vms. From there, you can ssh, halt and start up the VMs, e.g.:

cd ~/delivery-cluster/.chef/vms

build-node-test-1.vm  delivery-server-test.vm      Vagrantfile

vagrant halt
vagrant up
vagrant ssh build-node-test-1

Configuring the Workstation

In this section, we’re going to follow the Tutorial section 2 to create a Delivery organization and user.  But we’re going to short-cut a step by first generating an SSH key for the user. This only makes sense because we are both the user and Administrator of Chef Delivery on the workstation: the Tutorial splits the actions because that is more representative of normal use.

Generate an SSH Key for the Chef Delivery User

Generate an ssh key to use in your Chef Delivery user account:

ssh-keygen -t rsa -b 4096 -C "test@example.com"

I recommend saving it to /home/<user>/.ssh/delivery_rsa rather than the default id_rsa file. Do not enter a passphrase (press <Enter> twice when prompted).

Create or append the following in your ~/.ssh/config file, to make sure that the above key is used when communicating with the Delivery git server:

        IdentityFile /home/test/.ssh/delivery_rsa
        User test
        IdentitiesOnly yes

test is the name of the user we are going to create in Chef Delivery.

Create an Organization and User

Logon to the Delivery UI at:

using the admin user and password from the rake info:delivery-creds command.

Follow  Tutorial Section 2 to create the ‘delivery-demo’ organization and ‘test’ user. Set the user id to ‘test’ rather than ‘jsmith’. Specify the SSH public key at the same time as creating the user.

Your public key is here:

cat ~/.ssh/delivery_rsa.pub

Once the user is created, verify the setup and create a ‘known hosts’ entry by authenticating to Delivery git server.

ssh -l test@test -p 8989

The authenticity of host '[]:8989 ([]:8989)' can't be established.
RSA key fingerprint is 11:ce:26:01:b3:ee:f7:7f:4c:e5:ea:a5:91:a6:0d:6a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[]:8989' (RSA) to the list of known hosts.
Hi test@test! You've successfully authenticated, but Chef Delivery does not provide shell access.
                 Connection to closed.

If you get ‘Permission denied’, check that you have set the correct public key and user name in Chef Delivery, and the correct key file and user name in the ssh config file. Also check that the private key (~/.ssh/delivery_rsa) is only readable by the user (e.g. mode ‘0600’).

If you’ve previously installed Delivery,  you may get a warning because of a previous ‘known hosts’ entry for

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/test/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/test/.ssh/known_hosts:8
  remove with: ssh-keygen -f "/home/test/.ssh/known_hosts" -R []:8989
RSA host key for []:8989 has changed and you have requested strict checking.
Host key verification failed.

It is OK to remove the ‘known hosts’ entry using the ssh-keygen command given in the message, because you know you have created a new VM using the same address.

Configure Workstation to use Chef Delivery Server

We’re now going to setup a workspace for Delivery projects:

mkdir -p ~/delivery-demo
cd ~/delivery-demo
delivery setup --ent=test --org=delivery-demo --user=test --server=

The delivery setup command creates a .delivery/cli.toml file which is used by the delivery CLI whenever it is run in ~/delivery-demo or any subdirectory.

cat ~/delivery-demo/.delivery/cli.toml
 api_protocol = "https"
 enterprise = "test"
 git_port = "8989"
 organization = "delivery-demo"
 pipeline = "master"
 server = ""
 user = "test"

Create Acceptance Test Node

The last thing we need to do to configure the workstation is create the test node(s) for the demo application. The Tutorial only requires a single test node, in the Acceptance environment. Normally, you would need one or more nodes in each of the Acceptance, Union, Rehearsal and Delivered environments.

Create a Vagrantfile in ~/delivery-demo/Vagrantfile and copy the following into it:

Vagrant.configure('2') do |outer_config|
  outer_config.vm.define "acceptance-test-1" do |config|
    config.vm.network(:private_network, {:ip => ''})
    config.vm.box = "opscode-ubuntu-14.04"
    config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
    config.vm.hostname = "acceptance-test-delivery-demo-1"

Now use it to start a new Ubuntu VM:

cd ~/delivery-demo
vagrant up

We need to bootstrap this node and register it with the Chef server. The node needs to be in the acceptance environment for the delivery-demo project, which will be named ‘acceptance-test-delivery-demo-awesome_customers_delivery-master’ (acceptance-<enterprise>-<organization>-<project>-<pipeline>) .

cd ~/delivery-cluster
knife environment create acceptance-test-delivery-demo-awesome_customers_delivery-master
knife bootstrap --node-name awesome_customers_delivery-acceptance \
  --environment acceptance-test-delivery-demo-awesome_customers_delivery-master \
  --run-list "recipe[apt],recipe[delivery-base]" -xvagrant -Pvagrant --sudo
knife node run_list set awesome_customers_delivery-acceptance \

We bootstrap using the ‘delivery-base’ recipe, as this will install Chef push-jobs. We then set its runlist to include the ‘awesome_customers_delivery’ cookbook, as that is what we are testing. Note we cannot bootstrap with this recipe because it is not loaded into the Chef Server yet.

Following the Chef Delivery Tutorial

You should now be able to follow the Chef Delivery Tutorial, starting at the fourth section to Create a project.  Go through to the first step of  Step 8 (Deliver the Change) of that Section. In that step, you will watch your project go through the Acceptance Phase and then try to navigate to the application in acceptance test at It won’t be there.

What went wrong? First, I recommend reading the ‘Learn more about the deployment process’ foldout in Step 8. Then in the Delivery UI, look carefully at the Deploy stage. At the end you will see something like:

Recipe: delivery-truck::deploy
  * delivery_push_job[deploy_awesome_customers_delivery] action dispatch (up to date)

Running handlers:
Running handlers complete
Chef Client finished, 0/1 resources updated in 02 seconds

Basically, the push job that should have run chef-client on the acceptance test node did nothing. Why?

The delivery-truck::deploy recipe searches for nodes in the correct environment, with push-jobs and the project cookbook in their list of recipes.  Specifically, the search term used is "recipes:#{cookbook.name}*". This search term will match against the last-run recipes on the node, NOT against the recipes in the current run-list. When we bootstrapped the acceptance test node, we did not include the awesome_customers_delivery cookbook because it had not been uploaded to the Chef Server yet. The application cookbook was only uploaded in the Publish stage of the Verify phase. This is the ‘chicken-and-egg’ situation I referred to earlier.

To get round this, we will run a one-off manual push-job to converge the node:

cd ~/delivery-cluster
knife job start chef-client awesome_customers_delivery-acceptance

You should now be able to navigate to the application at Future runs through the pipeline will automatically run the push-job and what you will see in the Acceptance Deploy phase is:

Converging 1 resources
Recipe: delivery-truck::deploy
  * delivery_push_job[deploy_awesome_customers_delivery] action dispatch
    - Dispatch push jobs for chef-client on awesome_customers_delivery-acceptance

Running handlers:
Running handlers complete

Continue with  Step 8 where you click ‘Deliver’ in the Acceptance Stage of the pipeline. You should now be able to finish the remaining sections in the Tutorial.