This post describes how you can run Chef Delivery on a laptop, using Vagrant. My main intent is to give you a way to work through Chef’s Delivery tutorial if you do not have access to AWS – or if you are really lost using a Windows workstation. Use the AWS CloudFormation template in the Tutorial if you can – the approach in this post is more error-prone. You have been warned.
I’m going to assume you are reasonably familiar with Linux, Vagrant and using the knife command – and that you already have Vagrant and Virtualbox installed and working.
We’re going to spin up multiple (at least 4) virtual machines. I’m using an 8-core Ubuntu laptop with 16GB memory. If you are running on something much smaller, good luck.
Plan of Attack
The first part (the bulk) of this post, Setting up Chef Delivery, replaces the Install Chef Delivery on AWS section of the tutorial. In it, we’ll use the delivery-cluster cookbook to provision a Chef Server, Delivery Server and build node as virtual machines using Vagrant.
The second part, Configuring the Workstation, sets up a workspace to use with Chef Delivery. If refers to and partly replaces the second and third sections of the Tutorial.
The final part of this post, Following the Chef Delivery Tutorial gives you some pointers on working through the remaining sections of the Chef Delivery tutorial (the ‘meat’ of the tutorial) using this setup. In particular, the first time you send the demo application through the pipeline, it won’t quite work, due to an interesting ‘chicken and egg’ problem. I’ll explain why and what to do about it when we get there.
Setting up Chef Delivery
Prerequisites
- Vagrant
- Virtualbox
- ChefDK 0.15.5 or more recent (includes Delivery CLI)
- Git
- build-essential (Ubuntu) or comparable development library
If you already have ChefDK installed, please check its version and upgrade if needed:
chef --version Chef Development Kit Version: 0.15.15 chef-client version: 12.11.18 delivery version: 0.0.23 (bf89a6b776b55b89a46bbd57fcaa615c143a09a0) berks version: 4.3.5 kitchen version: 1.10.0
If it’s an earlier version, you may not have the Delivery CLI and you may encounter errors when using delivery-cluster.
You need the development environment library appropriate to your OS, e.g. for Ubuntu:
sudo apt-get install build-essential
(see Chef Delivery docs for other platforms)
Obtain Chef Delivery license
If you do not have a license already, you can obtain a temporary license that will let you use Chef Delivery for the tutorial.
Copy the license into the home directory on your workstation.
cp delivery.license ~
The delivery-cluster cookbook will handle putting the license onto the Delivery Server, once created.
Prepare to provision using delivery-cluster
Clone the delivery-cluster repo from Github:
git clone https://github.com/opscode-cookbooks/delivery-cluster.git ~/delivery-cluster
Run the following from within that repo to generate the provisioning settings to be used:
cd ~/delivery-cluster rake setup:generate_env
Accept the default of test for Environment Name and Cluster ID. Change the Driver Name to vagrant:
Global Attributes
Environment Name [test]:
Cluster ID [test]:
Available Drivers: [ aws | ssh | vagrant ]
Driver Name [aws]: vagrant
Accept the default SSH Username and optionally change the Box Type and Box URL to use Ubuntu (which is what I am using). The default Centos selection should also work.
Driver Information [vagrant] SSH Username [vagrant]: Box Type: [opscode-centos-6.6]: opscode-ubuntu-14.04 Box URL: [....]: https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box
Here’s the Ubuntu Box URL for easier copy/paste:
Take the defaults for all other settings. You can see all of the expected settings in the output JSON shown in the next Section.
Be aware this setup is slightly different from the Tutorial – specifically:
- the IP addresses used for the Chef Server, Delivery Server and build node(s) are ‘33.33.33.xx’ rather than ‘10.0.0.xx’
- the Delivery Enterprise name is ‘test’ rather than ‘delivery-demo’
The rake command will generate the environment in ~/delivery-cluster/environments/test.json. You can rerun the command or edit the file directly if you need to.
Update the environment to include FQDN
We need to make a manual update to the Delivery environment file to work around an issue with provisioning Delivery to vagrant. Without this workaround, various configuration files will incorrectly use an IP address of 10.0.2.15 when trying to communicate with the Chef Server or Delivery Server. To avoid this, we need to specify the FQDN to be used for these servers.
Edit the file ~/delivery-cluster/environments/test.json so that the FQDN is specified:
{
"name": "test",
"description": "Delivery Cluster Environment",
"json_class": "Chef::Environment",
"chef_type": "environment",
"override_attributes": {
"delivery-cluster": {
"accept_license": true,
"id": "test",
"driver": "vagrant",
"vagrant": {
"ssh_username": "vagrant",
"vm_box": "opscode-ubuntu-14.04",
"image_url": "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box",
"key_file": "/home/test/.vagrant.d/insecure_private_key"
},
"chef-server": {
"fqdn": "33.33.33.10",
"organization": "test",
"existing": false,
"vm_hostname": "chef.example.com",
"network": ":private_network, {:ip => '33.33.33.10'}",
"vm_memory": "2048",
"vm_cpus": "2"
},
"delivery": {
"fqdn": "33.33.33.11",
"version": "latest",
"enterprise": "test",
"license_file": "/home/test/delivery.license",
"vm_hostname": "delivery.example.com",
"network": ":private_network, {:ip => '33.33.33.11'}",
"vm_memory": "2048",
"vm_cpus": "2",
"disaster_recovery": {
"enable": false
}
},
"builders": {
"count": "1",
"1": {
"fqdn": "33.33.33.14",
"network": ":private_network, {:ip => '33.33.33.14'}",
"vm_memory": "2048",
"vm_cpus": "2"
}
}
}
}
}
Note: It may not be necessary to specify the FQDN for the build node.
Provision the Servers
To start provisioning run:
export CHEF_ENV=test rake setup:cluster
Watch it for a few minutes, to make sure there’s no early failure. You should see it create a Vagrant machine for the chef server and start to run recipes to install and configure the server.
If it’s going OK, go for coffee. Actually, go for lunch. This will take a while. There’s a lot going on… not only is it installing the Chef Server, Delivery Server and Build node, but also setting up credentials and certificates.
Sometimes a download will timeout and the provisioning run will fail part way through. If this happens, try rerunning it.
Hopefully you will come back and find a successfully completed Chef run. The last node provisioned should have been the build node.
Now we need to make sure it actually worked. Let’s start by getting the information that will let us logon to the Servers. Run the following rake command:
rake info:delivery_creds Username: delivery Password: XSomeGeneratedPassword= Chef Server URL: https://33.33.33.10/organizations/test Delivery Server Created enterprise: test Admin username: admin Admin password: +cAnotherGeneratedPasswordg= Builder Password: UtAndAnotherGeneratedPassword4= Web login: https://33.33.33.11/e/test/
You should be able to logon to both the Chef Server and the Delivery Server using the URLs and credentials provided by the rake command. You will need to confirm a security exception with the browser, as we’re using self-signed certificates.
If you’re using Firefox and have previously installed Delivery, you may need to clear the old certificates from the browser first. Firefox will give you Error code: SEC_ERROR_REUSED_ISSUER_AND_SERIAL. Go to ‘Preferences > Advanced > Certificates > View Certificates’ in the Firefox menu and delete the entries for 33.33.33.10 from the ‘Server’ and ‘Authorities’ tabs.
For a final smoke test, run the following knife command from within the delivery-cluster repo:
knife node status build-node-test-1 available
Chef Delivery uses push-jobs, and the above command lists nodes that are visible to push-jobs. If you do not see your build node(s) when you run the command, something has gone wrong (Chef server certificate problem, incorrect IP address, ….). Double check your environment file and try rerunning the rake command.
If you do make a significant change to the environment settings (e.g. changing the box type), I recommend destroying the virtual machines (see next section) and starting with a fresh clone of delivery-cluster, to remove cached provisioning information.
Managing the virtual machines
The delivery-cluster cookbook creates Vagrant specifications for the virtual machines in ~/delivery-cluster/.chef/vms. From there, you can ssh, halt and start up the VMs, e.g.:
cd ~/delivery-cluster/.chef/vms ls build-node-test-1.vm delivery-server-test.vm Vagrantfile chef-server-test.vm vagrant halt ... vagrant up ... vagrant ssh build-node-test-1 ...
Configuring the Workstation
In this section, we’re going to follow the Tutorial section 2 to create a Delivery organization and user. But we’re going to short-cut a step by first generating an SSH key for the user. This only makes sense because we are both the user and Administrator of Chef Delivery on the workstation: the Tutorial splits the actions because that is more representative of normal use.
Generate an SSH Key for the Chef Delivery User
Generate an ssh key to use in your Chef Delivery user account:
ssh-keygen -t rsa -b 4096 -C "test@example.com"
I recommend saving it to /home/<user>/.ssh/delivery_rsa rather than the default id_rsa file. Do not enter a passphrase (press <Enter> twice when prompted).
Create or append the following in your ~/.ssh/config file, to make sure that the above key is used when communicating with the Delivery git server:
Host 33.33.33.11 IdentityFile /home/test/.ssh/delivery_rsa User test IdentitiesOnly yes
test is the name of the user we are going to create in Chef Delivery.
Create an Organization and User
Logon to the Delivery UI at:
using the admin user and password from the rake info:delivery-creds command.
Follow Tutorial Section 2 to create the ‘delivery-demo’ organization and ‘test’ user. Set the user id to ‘test’ rather than ‘jsmith’. Specify the SSH public key at the same time as creating the user.
Your public key is here:
cat ~/.ssh/delivery_rsa.pub
Once the user is created, verify the setup and create a ‘known hosts’ entry by authenticating to Delivery git server.
ssh -l test@test -p 8989 33.33.33.11 The authenticity of host '[33.33.33.11]:8989 ([33.33.33.11]:8989)' can't be established. RSA key fingerprint is 11:ce:26:01:b3:ee:f7:7f:4c:e5:ea:a5:91:a6:0d:6a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[33.33.33.11]:8989' (RSA) to the list of known hosts. Hi test@test! You've successfully authenticated, but Chef Delivery does not provide shell access. Connection to 33.33.33.11 closed.
If you get ‘Permission denied’, check that you have set the correct public key and user name in Chef Delivery, and the correct key file and user name in the ssh config file. Also check that the private key (~/.ssh/delivery_rsa) is only readable by the user (e.g. mode ‘0600’).
If you’ve previously installed Delivery, you may get a warning because of a previous ‘known hosts’ entry for 33.33.33.11:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is 11:ce:26:01:b3:ee:f7:7f:4c:e5:ea:a5:91:a6:0d:6a. Please contact your system administrator. Add correct host key in /home/test/.ssh/known_hosts to get rid of this message. Offending RSA key in /home/test/.ssh/known_hosts:8 remove with: ssh-keygen -f "/home/test/.ssh/known_hosts" -R [33.33.33.11]:8989 RSA host key for [33.33.33.11]:8989 has changed and you have requested strict checking. Host key verification failed.
It is OK to remove the ‘known hosts’ entry using the ssh-keygen command given in the message, because you know you have created a new VM using the same address.
Configure Workstation to use Chef Delivery Server
We’re now going to setup a workspace for Delivery projects:
mkdir -p ~/delivery-demo cd ~/delivery-demo delivery setup --ent=test --org=delivery-demo --user=test --server=33.33.33.11
The delivery setup command creates a .delivery/cli.toml file which is used by the delivery CLI whenever it is run in ~/delivery-demo or any subdirectory.
cat ~/delivery-demo/.delivery/cli.toml api_protocol = "https" enterprise = "test" git_port = "8989" organization = "delivery-demo" pipeline = "master" server = "33.33.33.11" user = "test"
Create Acceptance Test Node
The last thing we need to do to configure the workstation is create the test node(s) for the demo application. The Tutorial only requires a single test node, in the Acceptance environment. Normally, you would need one or more nodes in each of the Acceptance, Union, Rehearsal and Delivered environments.
Create a Vagrantfile in ~/delivery-demo/Vagrantfile and copy the following into it:
Vagrant.configure('2') do |outer_config|
outer_config.vm.define "acceptance-test-1" do |config|
config.vm.network(:private_network, {:ip => '10.0.0.15'})
config.vm.box = "opscode-ubuntu-14.04"
config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.04_chef-provisionerless.box"
config.vm.hostname = "acceptance-test-delivery-demo-1"
end
end
Now use it to start a new Ubuntu VM:
cd ~/delivery-demo vagrant up
We need to bootstrap this node and register it with the Chef server. The node needs to be in the acceptance environment for the delivery-demo project, which will be named ‘acceptance-test-delivery-demo-awesome_customers_delivery-master’ (acceptance-<enterprise>-<organization>-<project>-<pipeline>) .
cd ~/delivery-cluster knife environment create acceptance-test-delivery-demo-awesome_customers_delivery-master knife bootstrap 10.0.0.15 --node-name awesome_customers_delivery-acceptance \ --environment acceptance-test-delivery-demo-awesome_customers_delivery-master \ --run-list "recipe[apt],recipe[delivery-base]" -xvagrant -Pvagrant --sudo knife node run_list set awesome_customers_delivery-acceptance \ "recipe[apt],recipe[delivery-base],recipe[awesome_customers_delivery]"
We bootstrap using the ‘delivery-base’ recipe, as this will install Chef push-jobs. We then set its runlist to include the ‘awesome_customers_delivery’ cookbook, as that is what we are testing. Note we cannot bootstrap with this recipe because it is not loaded into the Chef Server yet.
Following the Chef Delivery Tutorial
You should now be able to follow the Chef Delivery Tutorial, starting at the fourth section to Create a project. Go through to the first step of Step 8 (Deliver the Change) of that Section. In that step, you will watch your project go through the Acceptance Phase and then try to navigate to the application in acceptance test at http://10.0.0.15. It won’t be there.
What went wrong? First, I recommend reading the ‘Learn more about the deployment process’ foldout in Step 8. Then in the Delivery UI, look carefully at the Deploy stage. At the end you will see something like:
Recipe: delivery-truck::deploy * delivery_push_job[deploy_awesome_customers_delivery] action dispatch (up to date) Running handlers: Running handlers complete Chef Client finished, 0/1 resources updated in 02 seconds
Basically, the push job that should have run chef-client on the acceptance test node did nothing. Why?
The delivery-truck::deploy recipe searches for nodes in the correct environment, with push-jobs and the project cookbook in their list of recipes. Specifically, the search term used is "recipes:#{cookbook.name}*". This search term will match against the last-run recipes on the node, NOT against the recipes in the current run-list. When we bootstrapped the acceptance test node, we did not include the awesome_customers_delivery cookbook because it had not been uploaded to the Chef Server yet. The application cookbook was only uploaded in the Publish stage of the Verify phase. This is the ‘chicken-and-egg’ situation I referred to earlier.
To get round this, we will run a one-off manual push-job to converge the node:
cd ~/delivery-cluster knife job start chef-client awesome_customers_delivery-acceptance
You should now be able to navigate to the application at http://10.0.0.15. Future runs through the pipeline will automatically run the push-job and what you will see in the Acceptance Deploy phase is:
Converging 1 resources Recipe: delivery-truck::deploy * delivery_push_job[deploy_awesome_customers_delivery] action dispatch - Dispatch push jobs for chef-client on awesome_customers_delivery-acceptance Running handlers: Running handlers complete
Continue with Step 8 where you click ‘Deliver’ in the Acceptance Stage of the pipeline. You should now be able to finish the remaining sections in the Tutorial.
Graphs and Chef and Javascript - oh my! 