Wednesday, 24 June 2015

Force12 - past, present, future

Force12 is still a very new project but this post is on why we’re building it and where we see it going. At the moment it’s just a demo but this post describes how we’re going to get to our final goal.

We want Force12 to be an open source container scheduler that provides microscaling. Microscaling to us means providing QoS (Quality of Service) for your containers. So you run the right mix of containers at the right time. We see Force12 as specialising in QoS. So Force12 will integrate with existing schedulers like Marathon that provide fault tolerance.

Background -

Anne and I first started working together at Figleaves. During our time there one of the big changes was the move from physical to virtual machines. We followed a typical approach of virtualising dev and test first before finally migrating production.

We worked closely with the Ops team to help plan these migrations to minimise system and developer downtime. Finally we’d see the results; stacks of obsolete servers arriving at the office before disposal. These had been replaced by far fewer more modern servers running VMware.

Another big influence on Force12 is that Figleaves is a very seasonal business. The autumn would mean deploying final changes, capacity testing and a freeze on major changes. Peak would begin around Black Friday and continue until Valentines Day with Christmas and January sale in between. After Valentines Day the change freeze would be lifted.

Since we were running our own physical servers the capacity planning had to be for the busiest hour during peak. This would usually be on the last shipping day before Christmas. Using public cloud and auto scaling could have helped a lot.


After Figleaves I worked with Anne again on What’sMySize. What’sMySize provides personalised size guides for customers of fashion retailers. It’s developed in Ruby on Rails and is hosted on AWS using Elastic Beanstalk and RDS Postgres. One of the reasons we chose this architecture is because it supports auto scaling.

However auto scaling with VMs is difficult because it can take 4 – 5 minutes for an instance to boot and be joined into an Elastic Beanstalk cluster. This means you need to implement workarounds like scaling up early and scaling down slowly. Or auto scaling based on time periods and just running fewer servers during quiet periods.


Last year I’d started hearing about Docker and was following it from a distance. I’d also looked at the Docker support in Elastic Beanstalk. At the time it only supported 1 container per VM. This helped with deployment but that wasn’t a problem I had. The launch of ECS (EC2 Container Service) at re:Invent in November last year got me more interested.

In February at the Barcelona on Rails meet-up I saw a great presentation by Jorge Dias on using Docker with Rails. At this stage I wanted to try something out. So I dipped my toe in the water by moving by blog (MiddleMan / Ruby) from a Vagrant VM to using Docker and Compose.

This was followed by another post on running a Rails app on Docker in development. Lastly I did a Hello World example using ECS.

I’d been talking with Anne about Docker and ECS as I’d been writing these posts. We both thought that auto scaling was a great use case for containers because its possible to scale up and scale down in close to real time. We also thought that auto scaling worked well with micro-services architectures. As the time of day and current conditions affect the load on each service and their relative priority.

However nobody was really talking about auto scaling. There are good reasons for this because there is still lots to do on containers for security, networking, storage, fault tolerance, etc.

So we decided to build a demo on ECS, which we launched in May. It shows how quickly containers could be scaled against a random demand metric. There are posts describing what is Force12, the Force12 demo and the ECS cluster design.

Scaling Up

We launched the demo very early and at the moment its taking 3-4 seconds to start containers and under a second to stop them. We think we can reduce that to 1-2 seconds. However ECS is a new service and we’ve hit some problems including a bug with the ECS Agent because we’re constantly starting and stopping containers.

Once that bug is fixed we’re going to scale up the demo to support more containers. We’re also blogging about what we learn, here are posts on networking problems with CoreOS and how to setup New Relic monitoring on CoreOS.

More platforms

We started with ECS because we’re very familiar with AWS. This meant we could build the demo quickly. However we don’t see Force12 being tied to any specific platform. So we’re looking into running another demo on bare metal servers from and getting Force12 running on Mesos / Marathon.

Other platforms such as Kubernetes will follow. But with a small team we need to prioritise and we think Marathon is a good match to integrate with Force12.

Real metrics

To switch from being a demo to a usable product a key step is supporting real metrics. We intend to support a wide range but the first 2 metrics will be requests per second for load balancers and message queue length.

Currently the Force12 demo uses a REST API hosted on Heroku and auto scaled using AdeptScale. Another key step will be moving this API into the demo, the “eat your own dog food” approach.

Open source version

Once we have Force12 using real metrics and supporting multiple platforms we’ll make the code open source and publish the container image so you can run it.

Until then we’re going to continue to shepherd it as as closed source project. But we’re looking to expand our core team. So if you’re also exciting about microscaling and want to help develop Force12 please get in touch!

Monday, 8 June 2015

CoreOS networking problems on ECS

This post describes some problems we’re having with our microscaling demo on ECS (EC2 Container Service). It describes why we switched from Amazon Linux to CoreOS and why today we’ve switched back to Amazon Linux.

The problems we’ve been having with CoreOS are around networking. We still really like CoreOS and we’d like to resolve the problems. So any feedback or fix suggestions would be greatly appreciated. Please add them in the comments.

Move from Amazon Linux to CoreOS

At the moment Amazon Linux and CoreOS are the 2 choices of operating system for running container instances in an ECS cluster. We started out using Amazon Linux and were happy with it until a new AMI was released with v1.1 of the ECS Agent and Docker 1.6.

This was just before we launched and we saw a massive slowdown in responsiveness of the demo. At this point we switched to CoreOS stable as it gave us more control and allowed us to continue using v1.0 of the ECS Agent and Docker 1.5.

Fixing issue with v1.1 Agent

After we’d launched we worked with AWS to resolve the performance problems with the v1.1 agent. It turned out the problem was at our end because we weren’t trapping the SIGKILL signal

The 1.1 agent stops containers in a more correct manner but because we weren’t trapping the correct signals our demo containers were hitting a 30 second timeout of the docker stop command. So our containers were being force killed (status 137). We fixed our demo containers and upgraded to the v1.1 agent on CoreOS stable.

Trying CoreOS beta for Docker 1.6

Another benefit of the CoreOS switch was we got to try an operating system optimised for containers. Overall I like the CoreOS approach but there is a steep learning curve due to systemd and we’re hit several problems with it being a new distro.

One thing we’ve been keen to try is the upgrade from Docker 1.5 to 1.6. This moved from the alpha to the beta channel last week. At that point we switched our staging rig to CoreOS beta to see the effect on performance and stability.

DHCP with multiple interfaces

On CoreOS beta we continued to see networking problems. We would see the message below and see network connectivity problems from our containers. These could be the ECS Agent becoming disconnected, the Force12 scheduler being unable to call the ECS API or our demand randomizer being unable to call our REST API on Heroku.

kernel: IPv6: ADDRCONF(NETDEV_UP): veth01426c4: link is not ready

I looked for networking issues on the CoreOS bugs repository on GitHub. There wasn’t a direct match but this issue about DHCP with multiple interfaces seemed related.

Since our problems were with networking I decided to simplify our configuration by not assigning public IPs to our container interfaces. This turned out to be quite involved and meant using a lot of VPC features I hadn’t used before. Our new VPC setup has public and private subnets.  All the ECS container instances run in the private subnet. The public subnet has a NAT instance, which provides outbound connectivity to the ECS instances for applying updates. This guide was very useful in setting it up.

Back to Amazon Linux and turning off checkpointing

Removing the public interfaces seemed to help but we continued to have connectivity problems. So we decided to move back to Amazon Linux since our problems with the v1.1 agent are fixed and we’re seeing good performance with Docker 1.6. This worked well on staging but we spotted an issue when a container instance had 0 containers running. There was a pause of around 30 seconds and the message “Saving state!” appears in the logs.

This is a feature called checkpointing, which saves the state of an ECS agent so it can be restored if an instance crashes. However for our use case we don’t need it so we turned it off by setting the ECS_CHECKPOINT environment variable to false.

Current Status

Now we’re back on Amazon Linux we’re seeing good performance. We’ve also been able to turn up the demand randomizer to make it a bit spikier. This is part of our continued plans to scale up the demo as we learn more about ECS and improve the Force12 scheduler. However we’re still seeing periods of 30 to 60 seconds where the ECS Agent becomes disconnected. We’re continuing to investigate this, as it’s the biggest stability problem with the demo at the moment.

Thursday, 4 June 2015

Force12 demo architecture is a demo of microscaling containers using ECS (EC2 Container Service) from AWS. It shows containers being rapidly stopped and started based on a randomized demand metric. To use a networking analogy Force12 is providing QoS (Quality of Service) for containers.

In a router QoS will prioritise voice data over downloads because the VOIP traffic is more demand sensitive. With containers a public API used by a mobile app would be more demand sensitive and higher priority than a worker process performing a background task.

Our previous post described the demo in more detail. This post is on the design of the ECS cluster. A later post will be on the wider architecture which includes a REST API hosted with Heroku and the front end which is an Angular app.

Generally building the demo has gone well considering how much new technology we’re using. However there have been some problems and changes of direction and they are described in this post.

Force12 ECS cluster

EC2 Auto Scaling Group

The cluster consists of 3 m3.medium instances running in an Auto Scaling Group. We use m3.medium because it’s the smallest instance type that isn’t throttled like the t2 series. We use spot instances to keep the costs down.


In ECS terminology a Container Instance is a VM that is a node in the ECS cluster. For these VM’s there are currently 2 choices of operating system Amazon Linux or CoreOS.

We originally started using Amazon Linux but then switched to CoreOS. Mainly this was because I was interested to try an operating system optimized for containers. I’ve been impressed with CoreOS and especially their documentation. These pages on ECS, EC2 and Vagrant have all been essential. At the moment we’re running CoreOS stable. Now that Docker 1.6 is in the beta channel we’ll be switching to that.

For the demo we decided early on that we weren’t going to open source the project when we launched. Instead we wanted to launch a demo as early as possible to see what interest there was in the idea. For that reason we’re using a private repository from to host our containers.

This did cause some problems as initially we set up our ECS cluster in AWS’s EU (Ireland) region. Mainly we do projects for European clients and so we prefer to keep our data in the EU and close to their customers.

Since we don’t have any customer data on this project we moved everything to the US East region. Quay also seems to be hosted in US East and we got a noticeable increase in performance after the move. So we think the choice and location of your Docker repository is an important one. Since container launch speed is important for us we’re thinking about running our own repository bringing it even closer to our ECS cluster.

System Containers - ECS Agent & New Relic

The demo shows the containers running on our ECS cluster but it doesn’t show the 2 extra system containers installed on each node.

The ECS Agent is written in Go and it calls Docker to start and stop containers on behalf of the ECS Scheduler.

The New Relic container provides their server monitoring plus some extra metrics they’ve developed for Docker. The Docker socket running on the CoreOS VM is mounted in the New Relic container so it can be monitored.

I’ve written a post on how to install these containers as services when booting a container instance running CoreOS into an ECS cluster.

Force12 scheduler

The Force12 scheduler is written in Go. It polls a DynamoDB table to get the random demand metric. When the demand changes it stops containers to create capacity and then starts containers to match the desired quantity.

To stop and start these containers the scheduler calls the ECS API. In ECS terminology these are actually called tasks. Each task can have multiple containers but in our case each task has a single container.

The scheduler is written in Go rather than Ruby because we felt when we release it we’ll need the additional speed. The other reason is it’s my business partner Anne who does the scheduler development. She has a strong C background and so is much happier working with Go than Ruby.

Demand randomizer

The demand-RNG container is developed in Ruby. Its only responsibility is setting the random demand metric and updating the DynamoDB table. Both the force12 and demand-RNG containers run as ECS services. This means if a container dies it is replaced automatically.

Demo containers – priority1 & priority2

Our original idea for the demo was much more complex. We wanted the demo containers to be constantly generating a series of random visualizations. As we got into building the demo we realised this was over complex.

What we really wanted to show is that one of the properties of containers is ideal for autoscaling. Containers can be stopped and started in close to real time whereas with Virtual Machines this takes minutes. So the demo containers aren’t actually doing anything. They are based on the minimal busybox image and run sleeps of 1 second in an infinite loop.

However we did have a problem with the demo containers. We saw a big performance drop when we upgraded the ECS Agent from v1.0 to v1.1. The newer agent stops containers in a more correct way but this was causing timeouts when the agent calls the docker stop command.

The problem was we weren’t trapping the SIGTERM signal. This meant our containers were being force killed (Docker status 137) instead of stopping cleanly (Docker status 0). We got some great support from the ECS development team on GitHub who helped us find this.

Current Status

We’re still working on some issues with the current demo. The cluster mainly tracks the demand metric but we’re still seeing 30 second periods where the cluster stops responding.

A possible cause of this is the ECS Agent stops containers but it waits 3 hours before removing them. This is useful for debugging purposes but in our case it means up to 700 containers build up on each instance after 3 hours. So it could be a “garbage collection” problem when the stopped containers are being removed. The ECS team are working on an enhancement for the stopped containers issue and we’ve given our feedback on what works for us.

There are also several fixes / enhancements that we want to make to the front end. The front-end developers we usually work with are busy on other projects. We didn’t want to delay the launch so it meant I did most of the front-end development but I’m much happier working on the back-end and infrastructure parts of the stack.

What’s Next?

We’ve had some great reaction to the demo so we want to keep on showing that autoscaling is a great use case for containers. We’re also trying to do blog driven development as we make changes. For us getting people talking about micro scaling with containers is just as important as developing Force12.

For turning Force12 from a demo into an open source product the next step is to start using real demand metrics. At the moment our REST API is a Sinatra Ruby app hosted on Heroku and autoscaled using AdeptScale. We’re going to move that in house and host the API on the ECS cluster and auto scale it using CloudWatch metrics.

Other areas we’re looking at are scaling up the demo and moving it to other platforms such as Kubernetes or Mesos. We chose to develop the demo on ECS and the AWS platform for 2 reasons. We’re very familiar with AWS and we thought using AWS was the quickest way we could launch the demo. However the “batteries included, but removable” design approach is something we support and long term we don’t see Force12 being tied to a specific platform.