Tuesday, 19 May 2015

Monitoring Docker with New Relic on CoreOS and ECS

This post is on how to use New Relic server monitoring on ECS (EC2 Container Service) using CoreOS.

Update - 9 November 2015

This post is out of date as it was based on an early version of New Relic's monitoring for Docker. You can find an updated systemd unit file on GitHub (lorieri/coreos-newrelic). This worked for me with CoreOS stable 766.4.


Last Friday we launched our Force12 demo of container autoscaling. This is the first of a series of posts on how we're using ECS and what we've learnt from building the demo.

Docker Metrics in New Relic server monitoring


We're running our 3 EC2 instances in an Auto Scaling Group. The Launch Configuration for the EC2 instance installs 2 Docker containers as services. This reflects the architecture of CoreOS which is to keep the OS as minimal as possible and install extra components as containers.
  • ECS Agent - controls Docker on the EC2 instance and communicates with the ECS API
  • New Relic System Monitor - the New Relic sysmond service deployed as a Docker container
At the end of the post is our full cloud-config configuration. Its worth noting that the syntax has to be exact and issues like trailing whitespace will prevent the services being installed.

CoreOS with Quay.io Private Repository

The starting point for our CoreOS setup was their ECS example configuration. We're using private repositories from Quay.io so we configure this by adding the environment variables ECS_ENGINE_AUTH_TYPE and ECS_ENGINE_AUTH_DATA. To get the auth data run the docker login command on your local machine and use the data created in the .dockercfg file.

$ docker login quay.io

# .dockercfg
{"quay.io":{"auth":"***YOUR_AUTH_DATA***","email":"email@example.com"}}

New Relic

The extra Docker metrics for New Relic server monitoring are currently in beta. Initially when we installed the server monitoring it was working but there were no Docker metrics. This was fixed by adding this parameter -v /var/run/docker.sock:/var/run/docker.sock

This mounts the Docker socket running on the host in the New Relic container so it can monitor it. This forum post was very useful in getting this working and this issue is being worked on at New Relic.

cloud-config

Here is the full cloud-config file that runs when each EC2 instance is launched. To recap make sure you're setting the following:
  • YOUR_ECS_CLUSTER - make sure this matches your ECS cluster name.
  • YOUR_AUTH_DATA - set this if you're using private repositories.
  • YOUR_NEWRELIC_LICENSE_KEY - this should be your New Relic license key without quotes.
#cloud-config

coreos:
 units:
   -
     name: amazon-ecs-agent.service
     command: start
     runtime: true
     content: |
       [Unit]
       Description=Amazon ECS Agent
       After=docker.service
       Requires=docker.service

       [Service]
       Environment=ECS_CLUSTER=YOUR_ECS_CLUSTER
       Environment=ECS_LOGLEVEL=info
       Environment=ECS_ENGINE_AUTH_TYPE=dockercfg
       Environment=ECS_ENGINE_AUTH_DATA=YOUR_AUTH_DATA
       ExecStartPre=-/usr/bin/docker kill ecs-agent
       ExecStartPre=-/usr/bin/docker rm ecs-agent
       ExecStartPre=/usr/bin/docker pull amazon/amazon-ecs-agent
       ExecStart=/usr/bin/docker run --name ecs-agent --env=ECS_CLUSTER=${ECS_CLUSTER} --env=ECS_LOGLEVEL=${ECS_LOGLEVEL} --env=ECS_ENGINE_AUTH_TYPE --env=ECS_ENGINE_AUTH_DATA --publish=127.0.0.1:51678:51678 --volume=/var/run/docker.sock:/var/run/docker.sock amazon/amazon-ecs-agent

       ExecStop=/usr/bin/docker stop ecs-agent
   -
      name: newrelic-system-monitor.service
      command: start
      runtime: true
      content: |
        [Unit]
        Description=New Relic System Monitor (nrsysmond)
        After=amazon-ecs-agent.service
        Requires=docker.service

        [Service]
        TimeoutStartSec=10m
        ExecStartPre=-/usr/bin/docker kill nrsysmond
        ExecStartPre=-/usr/bin/docker rm nrsysmond
        ExecStartPre=/usr/bin/docker pull newrelic/nrsysmond:latest
        ExecStart=/usr/bin/docker run --name nrsysmond --rm \
          -v /proc:/proc -v /sys:/sys -v /dev:/dev -v /var/run/docker.sock:/var/run/docker.sock --privileged=true --net=host \
          -e NRSYSMOND_license_key=YOUR_NEWRELIC_LICENSE_KEY \
          -e NRSYSMOND_loglevel=info \
          -e NRSYSMOND_hostname=%H \
          newrelic/nrsysmond:latest
        ExecStop=/usr/bin/docker stop -t 30 nrsysmond

No comments:

Post a Comment