Deploying a multi-node application to AWS using Terraform and Chef

I’ve been wanting to try out Hashicorp’s Terraform for a while now. So I thought I would repeat the use case in my blog post Deploying a multi-node application to AWS using Chef Provisioning, this time using Terraform for provisioning the instances and Chef to configure the software on the nodes.

Setup AWS access

If you’re familiar with AWS, you need the following:

• AWS Client installed
• Default credentials configured in ~.aws/credentials that allow you to create and destroy resources in the region that you will use. I’ll be using us-west-2.
• A valid key-pair to allow SSH access into the region you will use. I’ll be using a keyname of “test2_aws”
• Port 3001 open for TCP traffic on the default VPC in the region you will use

If you are less familiar with AWS, follow the instructions in my previous blog post to:

• Prepare the AWS Client
• Setup SSH access into AWS. As we’re not using Chef Provisioning, when doing this step, you don’t need to do the step at the end to put the SSH key in the knife.rb file.
• Enable external access to the application

Getting setup with Terraform

Download and install Terraform following the instructions here.

Verify the version you have installed (0.6.3 at time of writing) using:

terraform --version

To see the full set of commands available, enter:

terraform

These commands are documented in the Terraform docs. There is also a decent Getting started guide.

Create a directory that we will do all our work in, and a subdirectory for the Terraform module that we will create:

mkdir -p ~/terra/example
cd ~/terra/example

Creating the AWS Instances

We are going to use Terraform to create two instances in AWS, one which will be our database server, one which will be our application server.

Create a file called ‘example.tf’ in the terra/example directory. This is our terraform configuration file, in which we will define the resources to be created.

example.tf

provider "aws" {
region = "us-west-2"
}

resource "aws_instance" "dbserver" {
instance_type = "t2.micro"
ami = "ami-75f8e245"
key_name = "test2_aws"
}

resource "aws_instance" "appserver" {
instance_type = "t2.micro"
ami = "ami-75f8e245"
key_name = "test2_aws"
associate_public_ip_address = true
}

Lines 1-3 tell Terraform to create the resources using the AWS provider, in the us-west-2 region. Note that in version 0.6.3, region is required and Terraform will not pick up defaults from ~/.aws/config. It will use default credentials in ~/.aws/credentials or from the standard AWS environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. You can also specify credentials by setting access_key and secret_key properties of the AWS provider in the Terraform configuration file.

Lines 5-9 define an AWS instance which will be our database server, using a t2.micro instance created from an Ubuntu 14.04 AMI, and configured with the default SSH keypair that we created earlier (replace ‘test2_aws’ with your key name if you called it something else). If you are using a different region than us-west-2, you will need to use the EC2 AMI finder to choose the right AMI for your region. Enter your region, ‘trusty’ and ‘hvm:ebs’ to find it (don’t use the ones with io1 or ssd stores).

Lines 11-16 create a second instance which will be our application server. We make sure it will have a public IP address by specifying ‘associate_public_ip_address’.

To get a preview of what Terraform will do with this configuration, run:

terraform plan

You will see output similar to:

The plan lists the two instances to be created and the values of the attributes that will be set on them. A value of “<computed>” means that the value will be determined when the resource is created. The end of the plan summarizes how many resources will be added, changed or deleted.

Note that the two resources are listed alphabetically. This is not necessarily the order in which the resources will be created. Terraform calculates this order based on implicit dependencies (we will see an example of this later) and explicit dependencies described using the depends_on resource property. Because there are currently no implicit or explicit dependencies between our resources, Terraform will actually create them in parallel.

To actually apply the plan, run:

terraform apply

You will see output similar to:

You should see that the two instances were created in parallel. The output also tells you how many resources were actually added, changed or deleted. It should also tell you where it stored the state of the infrastructure (a .tfstate file). You can use terraform show to review this state.

Terraform uses the state file when you change and reapply the configuration to locate the resources it previously created, so it can update them rather than creating more instances (although for some types of change, such as changing the instance type, it will destroy and recreate the instance). Terraform will refresh the state file from actual AWS state when it creates a plan, so that it does not rely on stale data. For example, if you destroy one of the instances using the EC2 console and then run ‘terraform plan’, it will show you that the instance needs to be created.

We’re now going to destroy the instances, because next we want to add Chef into the picture to act as what Terraform calls a ‘provisioner’. Unfortunately there is currently no way to run a provisioner on an existing resource (unlike in Vagrant, where you can run the ‘vagrant provision’ command), so you will need to destroy the instances whenever you add provisioners or change their configuration.

To destroy the configuration, run:

terraform destroy

You should see confirmation that both instances have been destroyed. Until you’re familiar with Terraform’s behaviors around instance lifecycles, and particularly if you have had any failed apply attempts, I recommend double checking that all instances have been destroyed using the EC2 console.

Setting up Chef

Install Chef DK

We will need the chef client and knife. I will be using ChefDK, which you can download from here here.

Verify it is working by typing:

chef --version

Setup a Hosted Chef account

The Terraform Chef provisioner requires a Chef server, so I am going to use Hosted Chef. You can sign up for a free account here: https://manage.chef.io/signup. Download and unzip the ‘getting started’ package, as we will need the credentials it contains later.

Setup the example application chef repo

We’re going to use the example provided with the knife-topo plugin as our application content.

First, download the latest release of the knife-topo repository and unzip it. Copy the ‘test-repo’ directory into your working directory, e.g.:

cp -R ~/Downloads/knife-topo-1.1.0/test-repo ~/terra

Then copy your knife settings and certificates (e.g.,
the contents of chef-repo/.chef in the “Getting Started” download) into
test-repo/.chef. Verify that you can connect to Hosted Chef by running:

cd ~/terra/test-repo
knife client list

You should see at least one entry for your workstation Chef client.

Upload cookbooks

We need to upload the cookbooks that are used to deploy the example application. To do this, we will use Berkshelf and the Berksfile provided in ‘test-repo’. Run the following commands:

cd ~/terra/test-repo
berks install
berks upload

This will install the remote cookbooks locally in the Berkshelf, then upload them to Hosted Chef.

Deploy the application using Terraform and Chef

Update example.tf

We are now ready to modify our Terraform configuration so that it uses Chef to deploy the application and its dependencies. To do this, we need to add a ‘provisioner’ section to our resources in ‘~/terra/example/example.tf’. Edit the ‘dbserver’ resource as follows, replacing values in angle brackets "" as described below:

resource "aws_instance" "dbserver" {
provisioner "chef" {
server_url = "https://api.opscode.com/organizations/<your-org-name>"
validation_client_name = "<your-client-name>"
validation_key_path = "~/git/chef-repo/.chef/<your-validator-key>.pem"
node_name = "dbserver"
run_list = [ "apt", "testapp::db" ]
connection {
user = "ubuntu"
key_file = "<full-path-to-key>/test2_aws.pem"
agent = false
}
}
instance_type = "t2.micro"
ami = "ami-75f8e245"
key_name = "test2_aws"
}

Lines 2-13 define how to run Chef to setup the node.

Lines 3-5 setup the connection to Hosted Chef, and should correspond to the values in knife.rb, although in knife.rb the validation key path is called ‘validation_key’. Note that if you have ChefDK version 6.2 or greater, you can use your client key (node_name and client_key properties in knife.rb) in place of the organization validator.

Lines 6 and 7 define the name for the node and its runlist.

Lines 8-12 define how to connect to the AWS instance using SSH, in order to bootstrap Chef and perform the chef-client run. The key_file should point to the keypair you created earlier.

Edit the ‘appserver’ resource as follows:

resource "aws_instance" "appserver" {
provisioner "chef" {
server_url = "https://api.opscode.com/organizations/<your-org-name>"
validation_client_name = "<your-client-name>"
validation_key_path = "~/git/chef-repo/.chef/<your-validator-key>.pem"
node_name = "appserver"
run_list = [ "apt", "testapp::appserver", "testapp::deploy" ]
attributes {
"testapp" {
user = "ubuntu"
path = "/var/opt"
db_location = "${aws_instance.dbserver.private_ip}"
}
}
connection {
user = "ubuntu"
key_file = "<full-path-to-key>/test2_aws.pem"
agent = false
}
}
instance_type = "t2.micro"
ami = "ami-75f8e245"
key_name = "test2_aws"
associate_public_ip_address = true
}

For the appserver node, in addition to the runlist we specify a set of initial attributes (lines 8-14). These will be set on the node with ‘normal’ priority. One of those attributes needs to be set dynamically to the IP address of the database server. Terraform allows us to do this using Ruby interpolation syntax and a dot notation to reference the resource property, as shown in line 12. The set of properties that can be accessed like this on an instance are described in the AWS Provider documentation under ‘Attributes Reference’.

The reference we have just created from the appserver resource to the dbserver attribute is an implicit dependency, which will be used by Terraform when determining the order in which it should create the resources.

The final thing we’ll do is output the public IP address of the application server, so we can test our application once it is deployed. Terraform lets us specify both inputs and outputs for a module. Add an output by adding the following to the end of example.tf:

output "address" {
value = "${aws_instance.appserver.public_ip}"
}

Review and apply the change

To review the planned change, run:

cd ~/terra/example
terraform plan

The output will look the same as before, because the resources are listed alphabetically and only show the resources and attributes that are directly handled by Terraform, not those handled by Chef.

Now apply the change:

terraform apply

Unlike previously where the instances were created in parallel, the dbserver instance is created and provisioned using Chef before the appserver instance is created. At the end of a successful apply, you should see something like:

The public IP address of the application server is listed at the end. You can use this to access the application:

http://:3001/

You should see a “Congratulations!” message and some information about knife-topo.

Failed Terraform applies

If a ‘terraform apply’ fails, it does not automatically rollback. Instead, when you rerun the apply, it will skip any completed resources, and it will destroy partially completed resources before recreating them.

However, Terraform does not invoke any provisioner actions as part of its destroy, so if the fail happens during the provisioner run then the nodes are still registered in Hosted Chef. This will cause the rerun to fail with a message something like:

You will need to manually clean up the node and client, as described in the following section.

Review the resource dependency graph

If you want to understand the order in which resources will actually be created, you can review the resource dependency graph:

terraform graph

This will give you output something like:

This tells you that there are three root nodes, for the two AWS instances and the AWS provider. It then tells you that the ‘appserver’ instance is dependent on the ‘dbserver’ instance, and the ‘dbserver’ instance is dependent on the AWS provider. From this, you can deduce that Terraform will setup the AWS provider, then create ‘dbserver’, then create ‘appserver’.

Cleanup the example

To destroy the instances, run:

terraform destroy

This DOES NOT destroy the nodes in Hosted Chef. You need to do this separately, either through the UI or using the knife command:

cd ~/terra/test-repo
knife node destroy dbserver
knife client destroy dbserver
knife node destroy appserver
knife client destroy appserver