It's a live, real time view of containers being created and destroyed to meet demand on our AWS cluster.
Container Demand = red, Containers = dark blue.
It’s deliberately simple.
- There's a fixed pool of resources (3 EC2 VMs).
- 2 generic container types: Priority 1 (dark blue) and Priority 2 (lilac).
- 1 Demand_RNG container, which randomly generates demand for Priority1 containers (demand in red).
- 1 Force12 scheduler, which monitors demand and starts and stops Priority1 and Priority2 containers.
The second chart shows historical snapshots of the last few seconds (red demand, dark blue P1 containers and lilac P2 containers).
Below the charts you can see what’s happening now on each container instance.
GoalForce12’s job is to
- Meet demand by starting or stopping Priority1 containers.
- Use any leftover resources for Priority2 containers.
Success is when the running Priority1 containers meet the demand AND the total number of running Priority1 + Priority2 containers = 9 (maximum utilisation of fixed resources).
So if there are 4 Priority1 containers running there should be 5 Priority2 containers running. If the demand for Priority1 services increases by 1, Force12 will stop a Priority2 container and start a new Priority1 container ASAP.
The ResultsAs you can see from the demo, on a fairly untuned environment you can get container instantiation speeds of around 3 seconds (with some normal-style spread around that). Shutdown is much faster. This untuned instantiation time is far higher than the sub-second startup times we know are possible in a more tuned environment.
To improve stability and speed we’ll evolve this basic set-up and we’ll blog about what we do and what effect it has.
Known Issues with the Demo
- Demo container start time is 3-4 seconds. We want to achieve sub second speeds without heroic measures, i.e. with a standard cloud service setup.
- The container instances can become unresponsive to starts and stops and they can take 30 seconds to recover (but they do recover). This seems to be related to the container networking. We're looking into this.