Getting started with Chef 🤖
Having to migrate an app to a new server I asked myself, what is the best option to host and maintain my apps. Here is what I found out.
Let’s say you need this setup for running your new carefully crafted app:
- Mailserver (Spamfilter, imap, postfix..)
- Ruby, Python, nodejs
- PostgreSql, Redis, memcached
- Resque or Sidekiq
- DB and File Backup
- Staging and production environments
- ca. 4GB Ram
- 200GB SSD
Market: Comfort, Flexibility and Money 💸
For me there are two different approaches to meet these requirements.
1. Take something like heroku…
and add a service for everything. Get an email account at fastmail, send emails with mailgun, add a database option, a file server like S3, add a redis server and probably something else. So for Heroku this would cost 50$, plus redis 10$, add S3, Postgres 50$ and so on. Before taking this road I recommend carefully checking the pricing for each service. For this article I checked Mailgun and wondered about the free plan with 10.000 messages included. If you look closely you will recognize, it is only free until you hit 10.000 messages total, after that it is 79$/Month.. So it is basically like the 30days trial versions from back in the days. There are probably services that cost less, but it will not get cheaper as hosting your own server.
Let’s be nice and assume 200$ for option one.
- Easier to setup
- Powerful services for many tasks
- Not as flexible as option 2
- Mucho mucho expensive
2. Manage your own server
I usually use virtual servers, they are cheap and powerful enough for me. Now you have it, the freedom and power to do everything. But with power there comes responsibility. Managing a production server can be tough.
You can find virtual servers that meet the requirements for 20$.
- As flexible as it can get
- More difficult to setup
- Need to be maintained
Managing your Server
When you get your new server, you probably want to start quickly setting it up. But wait a minute. No matter if you have a virtual server or a real server, you cannot change the hardware. For a virtual server that often means, that after 5years (for Debian or Ubuntu LTS) you will have to setup everything again. Or maybe you need to scale an setup another system. Depending on your requirements there might be easier ways (maybe Docker containers), but what I want to write about is automated system configuration.
There are several prominent tools, Chef, Puppet and Ansible to name a few. I am using Chef for the examples. What all these solutions have in common, is that your infrastructure/server config is defined by code. You can check this code in to your repository and easily use it to setup a new server in no time.
Chef is a little hard to get started, but I found a good minimal setup. In the documentation you will see ‘Chef Workstation’, ‘Chef Client’ and ‘Chef Server’. I only use the client that lives on the server and a tool called ‘knife solo’.
gem install knife-solo gem install knife-solo_data_bag knife solo init mychefrepo
These commands build your infrastructure project! For testing you can spin up a vagrant server..
If you are using vagrant, make some adjustments to your Vagrantfile:
config.vm.hostname = "yourwebsite.com" config.vm.network "private_network", ip: "172.17.33.10"
It makes sense to have a special user on your server for running chef:
vagrant ssh adduser chef usermod -aG sudo chef visudo chef ALL=(ALL) NOPASSWD:ALL
Then add your ssh key to
Next let’s install the chef client on your server!
knife solo prepare firstname.lastname@example.org
This should install the chef client on your server and you are ready to go!
But what are all the files and folders in your project you might ask. Those are the important ones:
| |- cookbooks.. |- data_bags.. |- nodes.. |- roles.. |- site-cookbooks |- your_cookbook |- recipes.. |- templates.. |- metadata.rb |- Berksfile
The first cookbooks folder is for cookbooks that you get from the supermarket. That’s right, there is a supermarket, I will get to that in a minute.
data_bagsis for storing json data that you can use in your receipts.
nodesare your servers and specify what should run on which server.
site-cookbooksis where your code resides
meta_data.rbmucho important, you have to add your dependencies here.
Berksfile, like a Gemfile to install cookbooks from the supermarket.
So how would you install a database like Postgresql using this setup?
First go to the
site-cookbooks folder and add a new folder with your cookbook name, e.g. ‘cooking_noodles’
Also add the other folders recipes, templates and the metadata.rb file.
Under the folder
recipes add a file
So add the following line to your
cookbook 'postgresql', '~> 7.1.1'
And - very important - add this line to your
depends 'postgresql' ## https://supermarket.chef.io/cookbooks/postgresql
Now edit your
default.rb to install PostgreSQL and add some database:
postgresql_client_install 'My Postgresql Client install' do version '10.0' end postgresql_server_install 'Postgresql Server' do version '10' setup_repo false initdb_locale 'en_US.utf8' password 'your_secret_password_should_not_be_here' end postgresql_user 'some_user' do password 'your_other_secret_password_should_not_be_here' end postgresql_database 'some_database' do locale 'en_US.utf8' template 'template0' owner 'some_user' end
As I wrote, you should not check in your secret passwords in plain text. In chef there are encrypted data_bags for this use case!
There are different ways to approach it, here is how I do it..
First you need a text file that contains a strong encryption key. You use this file to encrypt or decrypt the file, also on your server.
Put this file in your project but do not check it in your repository!
knife solo data bag create passwords db --secret-file 'YOUR_SECRET_FILE_PATH'
Edit the file and add the secrets you need by following these instructions.
Then you have to put your secret key somewhere on your server. After that you can use your secrets in your receipts:
passwords = data_bag_item('passwords', 'db', IO.read('SECRET_FILE_PATH_ON_SERVER').strip) postgresql_user 'some_user' do password passwords[:YOUR_DATABASE_PASSWORD] end
Ready to go! All that is left to do is cooking your recipe. In your
nodes folder is a JSON file with your server ip. Open it up and edit the
"run_list": [ "recipe[cooking_noodles]" ]
This tells the chef client to run your default recipe. Time to cook:
knife solo cook email@example.com
This should install PostgreSql and add a database on your server. From there on you can add more recipes to your cookbook, add it to your run list, add more cookbooks from the supermarket until you have all configured.
Once you are ready you can run your configuration on as many servers as you like or adapt it for another app. But for me doing the server setup this way has even more advantages:
- you write configuration in Ruby
- you have an overview of your infrastructure in code
- with some adjustments it should even work on different systems, Ubuntu, Debian, CentOS etc.
I hope this little walkthrough worked out for you and helped to get you started. For me this was the most difficult thing, as there are so many different existing tools and products within Chef that had to be ruled out.
Let me know if something was unclear, there is always room for improvement.