Jesse's Software Engineering Blog
Amazon’s OpWorks allows for easy EC2 provisioning with Chef. Chef allows instances to configure all services, users, setting, dir/files, etc. as the instance is spun up. This allows for rapid infrastructure expansion and a consistent application stack.
Before beginning, be sure there is a key pair associated with the account, which will allow access to the EC2 instances. This can be done under EC2 -> key pairs. It is important to create the keys before starting as new keys cannot be added to an instance once the instance is running.
Setting Up OpWorks
Under OpWorks, select Add Stack. A stack is an entire application, containing various different layers which are all provisioned the same with multiple EC2 instances per layer. For this example the stack will be named jessesnet with Ubuntu 14.04 LTS for the OS. Be sure to use a vpc and the key pair created earlier. Under the advanced section, select the version of Chef to use and turn on custom cookbooks. In order for AWS to read custom cookbooks, the cookbooks will need to be accessible via a version control repo i.e. email@example.com:jessecascio/cookbooks.git. For the SSH key, use the private key of a public key associated with GitHub. Also activate Berkshelf. The custom JSON field is useful when custom parameters need to be passed into the cookbooks/recipes.
Although creating Chef cookbooks is outside the scope of this article, there should be at least one cookbook used with OpWorks for this example. The cookbook will be created using the Chef development kit.
Download and install the development kit on Ubuntu. Also, verify other dependencies are available
wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chefdk_0.3.5-1_amd64.deb sudo dpkg -i chefdk_0.3.5-1_amd64.deb sudo apt-get install ruby # verify gem gem -v sudo gem install test-kitchen sudo gem install busser # verify chef --version kitchen --version
Test kitchen is a robust testing framework used with Chef cookbooks, but again outside of the scope of this article. When developing Chef cookbooks, the test kitchen tutorial should be read through to gain understanding of it’s usefulness.
Instead of using kitchen for the cookbook generation, the chef program can create the default folder structure. Within the cookbooks GitHub folder
cd cookbooks chef generate cookbook jessesnet
This will create the folder structure needed for the Chef cookbook. In the default recipe, define Apache install instructions
vim jessesnet/recipes/default.rb package 'apache2' service 'apache2' do action [:start, :enable] end
Verify the recipe’s syntax
sudo chef-client --local -o recipe['jessesnet::default']
NOTE: Again, use Test Kitchen when creating Chef cookbooks. Test Kitchen offers various testing frameworks including Vagrant to run the recipes. This will save a lot of time as OpWorks takes a while to load recipes and having to wait to test is not optimal.
Now that the recipes are written and committed, we are able to provision the OpWorks layers. A layer contains numerous EC2 instance with the same functionality i.e. web servers, database servers, etc. Create a custom layer called web-server, with a short name of http. Under the settings of the layer, the security groups for the entire layer can be defined. For the web-server this would be to open ports 80 and 443.
Under the settings -> recipes for the layer, click Edit. The cookbooks can be defined on a variety of different lifecycle events during the EC2 life span. For example, the Configure event is triggered every time an instance is added or removed, allowing for easy maintenance as instances change. The default cookbook can be defined on the setup event by adding
On the main Instance tab, select Add Instance to the web-server layer, a micro is the cheapest. Notice the server name is the short name with an integer appended. OpWorks automatically updates the /etc/hosts file everytime a new instance is added or removed and maps both the public and private IP addresses to those host names which can be useful when creating recipes.
NOTE: Starting OpWorks instances take considerably longer to start than normal EC2 instances, usually ~10-15 mins.
After the server is booted SSH in and verify that Apache2 was installed and running. Can also verify via the web browser if the correct ports were opened via the security groups.
Another useful feature is the ability to manually run recipes on any number of instances. This can be done by selecting the stack tab and clicking on Run Command. Under the command drop down select Execute Recipes and pick the instances in which to run the recipes on.
NOTE: The OpWorks agent caches the cookbooks on each instance. If changes are pushed to the GitHub repo the instances will need to manually have their caches reset before those changes will be available. This can be done with the Update Custom Cookbooks command.
While not an in-depth tutorial on writing Chef cookbooks for OpWorks provisioning, this should be enough of an overview to demonstrate the power of AWS OpWorks. With a robust Chef cookbook library, managing an AWS infrastructure becomes much easier. EC2 instances are automatically provisioned, updates can be manually pushed to all of or particular sections of a cloud server cluster, and servers can be managed in bulk.