Grubhub uses a microservices architecture which requires developers to manage a wide array of various services in their development environment. To tackle this problem, Grubhub developed a vagrant-managed virtual machine using docker-compose to manage and run service dependencies. Utilizing this system, called garcon-in-a-box, we have had new developers attend orientation on Monday, learn our services and set up their development environment on Tuesday, and have their first code change ready for review Wednesday morning.
Grubhub, like many others, leverages the microservice architecture for stability, scalability, and separation of concerns. However, these microservices have ever-increasing numbers of service dependencies required to run them. These service dependencies are other services that get called in the operation of the service under development. At Grubhub, we depend on services we have written in-house and other open source utility services such as Cassandra, Eureka, and Elasticsearch. Local development while using many different service dependencies can be quite a challenge. How does one manage different services with different requirements running on all different types of developer machines?
Priorities for local development
To enable quick development, we needed to make a local solution to start up a series of services. When we started this effort, we identified a few priorities:
- It needs to handle as many use cases as possible
- It shouldn’t break often, and if it does, it should be easy to fix.
- It needs to be as easy as possible to start up for the first time, as starting the dev environment is often the first task of new developers, and we want to make them feel welcome
- It needs to be easy to configure the set of services and dependencies that you need for your use case, and not have any start up that you don’t care about
Docker-compose and Vagrant
Fortunately, the services we build at Grubhub are deployed as docker containers, so starting up services is easy as long as you use docker. To set up services with all the necessary properties and to facilitate setting up sets of services, we decided to use docker-compose. Using the docker compose file format makes it easy to specify the configuration necessary to run different services, so starting services becomes as easy as specifying a docker-compose file to run.
Docker-compose made it easy to run individual services, but we didn’t stop there. We also wanted to make it easy for teams to specify sets of services they used together, and run those as easily as possible. To accommodate this, we created a convention of shell scripts which simply reference a few docker compose configuration files. We also created a mechanism to specify which script you’d like to run on startup, to help keep the solution easily configurable per team.
At the time, docker only ran natively on linux machines, and our developers use a mixture of different OSs for their development machines, so that necessitated using a VM. To create the VM, we use Packer, and for our developers to run the VM in a reproducible way across their different host OSs, we use Vagrant. We called this VM garcon-in-a-box, named after our Grubhub’s internal cloud service framework, garcon.
To VM or not to VM?
When docker released docker for Mac in the middle of 2016, we were able to move to a non-VM solution. This forced us to consider what the benefits of removing the VM from our development environment are. The benefits are simplicity in the networking model and potential efficiencies in running a system of services due to the networking changes as well as not having to run in virtualization.
There are some drawbacks to moving outside of virtualization. Some networking changes need to be done on one’s development machine to allow you to connect to some service dependencies at the expected IP ranges. Additionally, each machine needs some common dependencies installed on them. This could be done via a set of scripts, but the scripts would have to be written to support all the different OSs that developers use at Grubhub.
Vagrant and VMs are a natural solution to these problems. Vagrant makes it as easy to start a development environment as installing Vagrant and VirtualBox and running vagrant up. This works no matter the host OS. If problems arise in the VM, bringing it to a clean state is as simple as vagrant destroy. If something goes wrong when developers have their services running on their host machine, it is much harder to fix, and the problem is compounded over different OSs, different versions of the service configurations, and different versions of the initial dependency setups.
For these reasons, we decided to stick with garcon-in-a-box. After a few months of dedicated development work, we had a system which was able to address our priorities. It handled all garcon use cases right out of the box, and can also support both older services we have running outside of docker as well as the new services that we are constantly building. Due to the dedicated development time, as well as continued patches, garcon-in-a-box’s stability is always increasing — and when users have a problem they can quickly restart their VM’s state with vagrant destroy, and quickly set up their services again using docker-compose and the shell scripts made to run a set of docker-compose configurations. Beginner start up is a three step process:
- Make sure credentials are set up for the new user in our internal systems.
- Install Vagrant and VirtualBox.
- Run vagrant up.
The microservice architecture offers a lot of benefits for development and operations, but how one creates new functionality in this paradigm is a topic that is worth thinking about. When a development system works well and is properly invested in, it can make the tedious task of developer environment setup painless.
After putting in a bit of effort on our development environment we had a flexible, configurable, and easy to start and fix system, which developers like more than previous systems. Most importantly, this allowed developers to stop worrying about their development environment and move on to exciting development on features and service improvements.