Running Marathon and Mini Mesos as containers

Mini Mesos is a great project recently released by Container Solutions for testing Mesos frameworks. It sets up a full Mesos cluster running inside a single Docker container. The cluster runs Zookeeper, Mesos Master and a configurable number of Mesos Agents. It also uses DIND (Docker in Docker) so that Docker containers can be run within the cluster.

The initial use case for Mini Mesos is for testing Mesos frameworks. So it’s designed to be run with Java as part of a JUnit test suite. However the Docker image has also been extracted to the mesos-local project.

At Force12.io we’re currently porting our microscaling demo from EC2 Container Service to Mesos and it will run on top of the Marathon framework. We’ve recently released coreos-marathon, which builds a 3-node Marathon / Mesos cluster locally with Vagrant or deployed to physical servers at packet.net.

This is great for testing and production but its quite heavy for development. So when we heard about Mini Mesos we were interested in integrating it into our development process. Here are the steps for setting this up.

Create Docker Machine

If you’re running OS X or Windows you first need to use Docker Machine to create a boot2docker VM for running the containers.

$ docker-machine create -d virtualbox --virtualbox-memory 4096 mini-mesos
$ eval $(docker-machine env mini-mesos)

Docker Compose

Next use Docker Compose to link the Marathon and Mini Mesos containers. The privileged flag needs to be true because we’re using DIND (Docker in Docker).

# docker-compose.yml

marathon:
  image: mesosphere/marathon:v0.9.0
  command: "--master zk://mesos:2181/mesos --zk zk://mesos:2181/marathon"
  links:
    - mesos
  ports:
    - "8080:8080"
mesos:
  env_file: .env
  privileged: true
  image: containersol/mesos-local
  ports:
    - "5050:5050"
  expose:
    - "2181"
  volumes:
    - "/sys/fs/cgroup:/sys/fs/cgroup:rw"

The .env file sets the necessary environment variables. In this case we’re running 2 Mesos Agents but this is configurable.

# .env

NUMBER_OF_SLAVES=2
MESOS_QUORUM=1
MESOS_ZK=zk://localhost:2181/mesos
MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
MESOS_CONTAINERIZERS=docker,mesos
MESOS_ISOLATOR=cgroups/cpu,cgroups/mem
MESOS_LOG_DIR=/var/log
SLAVE1_RESOURCES=ports(*):[9200-9200,9300-9300]
SLAVE2_RESOURCES=ports(*):[9201-9201,9301-9301]

Accessing Marathon & Mesos

You start the containers using Compose and use Machine to get the IP address of the boot2docker VM. The web UI’s for Mesos Master and Marathon are available on this IP address.

$ docker-compose up
$ docker-machine ip mini-mesos
192.168.99.100

• Mesos UI: http://192.168.99.100:5050
• Marathon UI: http://192.168.99.100:8080

Conclusion

Thanks again to Container Solutions for developing and releasing Mini Mesos. I can see it being useful for lots of projects. In our case it means we can develop our custom containers using Docker and have local access to a full Marathon / Mesos stack.