Automated Sensu Client Decommissioning

How to automatically decommission Sensu clients on AWS.

Posted by Abdel Kamel on August 8, 2015

The problem

If you are using Sensu, then you know that when you terminate an instance on AWS is going to trigger Keep-alive alerts. So one solution is to have a script that keeps polling AWS to check for shutting down or terminated instances then make a API request to Sensu for removal. But thats too much work for such a simple task.

An easier solution is to have Sensu trigger the removal through a handler. In my opinion, this is a much cleaner and easier approach. This set up is similar to the one posted on github here with some differences.

The difference is in the script. It matches clients on AWS with Sensu based on there private IP address.

Setting up the handler

We attach the awsdecommission handler to our default handlers. Notice that our default handlers is a set. So you can add more handlers (ie. slack).

default.json

{
  "handlers": {
    "default": {
      "type": "set",
      "handlers": ["awsdecommission"]
    }
  }
}

awsdecommision.json

{
  "handlers": {
    "awsdecommission": {
      "type": "pipe",
      "command": "/etc/sensu/handlers/awsdecomm.rb",
      "severities": [
        "ok",
        "warning",
        "critical"
      ]
    }
  }
}

Decommission script

In the script we require the aws-sdk library to query aws. However, there are 2 versions of the library. We are using version 2.

gem install aws-sdk -v '~> 2'

There is a list of region the script loops over. At the moment, it’s set only to one region us-west-2. You can add more regions like so %w{ us-west-2 us-east-1 }.

awsdecomm.rb

#!/usr/bin/env ruby

require 'sensu-handler'
require 'aws-sdk'

class Decomm < Sensu::Handler

  def delete_sensu_client
    puts "Sensu client #{@event['client']['name']} is being deleted."
    if api_request(:DELETE, '/clients/' + @event['client']['name']).code != '202'
      puts "Sensu API call failed"
    end
  end

  def check_aws
    %w{ us-west-2 }.each do |my_region|
      ec2 = Aws::EC2::Resource.new(region: my_region)
      my_instance_addr= @event['client']['address']
      not_found = true

      ec2.instances.each do |instance|
        if my_instance_addr.eql?(instance.private_ip_address)
          not_found = false
          puts "Instance #{@event['client']['name']} exists; Checking state"
          if instance.state["name"] != "running"
            puts "Instance #{@event['client']['name']} is #{instance.state["name"]}; I will proceed with decommission activities."
            delete_sensu_client
          end
        end
      end
      if not_found
        puts "Couldnt find #{@event['client']['name']} on AWS. Decommissioning instance. "
        delete_sensu_client
      end
    end
  end

  def handle
    if @event['action'].eql?('create')
      check_aws
    end
  end

end

Setup IAM credentials

Notice in the script we don’t reference a ACCESS_KEY_ID or a SECRET_ACCESS_KEY. We rely on an IAM role that presumably the Sensu server is attached to. If your Sensu server isn’t attached to a role, I highly recommend you bring online a new server with an attached role.

The role should look something similar to this.

{
  "Version": "2015-08-08",
  "Statement": [ {
    "Sid": "Stmt1438721067000",
    "Effect": "Allow",
    "Action": [ "ec2:Describe*" ],
    "Resource": [ "*" ]
  } ]
}

If for some reason you can’t use IAM roles then with small edits to the script and the handler you can pass a key and secret. Check out the documentation for details.



comments powered by Disqus