Tuesday, 17 May 2016

Microscaling Marathon with DC/OS on Azure Container Service

This post shows how you can use our Microscaling Engine with the Marathon container orchestrator. Microscaling makes a system's architecture dynamic – able to respond to real demand in real time. In this demo you'll scale a queue to maintain a target length.


To keep things simple we haven't set up service discovery on the Mesos cluster. Instead we're using an Azure Storage Queue that all the containers can access.



For the demo we'll be creating 4 Marathon Apps that run as Docker containers.
  • Producer - containers add items to the queue. We start 3 producers on startup and you can scale these manually using the Marathon UI.
  • Consumer - containers remove items from the queue and are scaled by the Microscaling Engine.
  • Remainder - any spare capacity is used for this background task. You can change the Docker image to use your own using the Marathon UI.
  • Microscaling - the engine scales the demo and sends data to our Microscaling-in-a-Box site.
For this walkthrough we'll be using a DC/OS cluster running on Azure Container Service. However the Microscaling Engine will integrate with any instance of Marathon.




Install Steps

Azure Storage Account

You'll need an Azure Storage Account and access keys to create the queue.

  • If you don’t currently have an Azure subscription you can get a free-trial.
  • Sign in to the Azure Portal.
  • Navigate to New -> Data + Storage -> Storage Account
  • Create a storage account with these settings.
  • Name - this must be globally unique across all Azure Storage Accounts.
  • Replication - make sure you choose Locally-redundant storage for the queue.
  • Resource Group - create a new resource group for the queue.
  • Once the storage account has been created navigate to Settings -> Access Keys and make a note of your access keys.

Launch ACS Cluster

  • See this tutorial for launching an ACS cluster.
  • In Basics screen set the following
  • USERNAME - e.g. "azureuser". You'll need this for connecting to the cluster via SSH.
  • SSH PUBLIC KEY - also needed for SSH access.
  • RESOURCE GROUP - choose a name for your cluster. 
See this tutorial for generating an SSH key if you don't have one.
  • In ACS Settings screen set the following

  • AGENT COUNT - increase from 1 to 2.
  • DNS PREFIX - choose a name for your cluster.

Open SSH tunnel

  • To access the Marathon REST API and the DC/OS web UIs you need to open an SSH tunnel on port 80 to the master node.

Create Marathon Apps

  • You're now ready to install the Marathon demo apps. Please sign up for a Microscaling-in-a-Box account if you don't already have one.
  • On the Start page choose Mesos / Marathon.
  • The Install page links to this post so you can continue to the Run page.

  • On the Run page you'll see the commands for installing the Marathon Apps. The marathon-install command needs Ruby but has no other dependencies.

  • For MSS_MARATHON_API use http://localhost/marathon if you're using ACS.
  • You'll need to also set AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY with the access keys you created earlier.

View The Demo

  • Once Marathon has launched the apps the results will appear in the Microscaling-in-a-Box site. You'll see the Microscaling Engine scaling the consumer and remainder containers to maintain the target queue length.
  • You can also view the demo in the DC/OS UI using the SSH tunnel. The DC/OS dashboard (http://localhost/), Marathon (http://localhost/marathon) and Mesos  (http://localhost/mesos) UIs are all available.

Cleaning Up

Uninstall the Marathon Apps

  • You can use the marathon-uninstall command to remove the demo apps from your cluster.
MSS_MARATHON_API=*YOUR_MARATHON_API* ./marathon-uninstall

Delete the Azure Resources

Important: You'll be charged for running the Azure VMs and any data stored on the Azure Queue. So its important to delete these when you don't need them anymore.
  • Sign in to the Azure Portal.
  • Select Resource Groups from the left hand menu.
  • Find and delete the 2 Resource Groups you created for the ACS cluster and the Azure Queue.

Next Steps

As well as supporting Marathon and Azure Storage Queues we also support the Docker API and NSQ. We'll be adding support for more queues and container orchestrators.

Stay abreast of all the latest developments by following us on Twitter or starring us on GitHub: