Treasure Agent Monitoring Service: Quickstart

Table of Contents

Overview

This article shows how to set up the monitoring service for Treasure Agent. The monitoring service provides the ability to:

  1. visualize and monitor Treasure Agent’s resource usage with a per-plugin level granularity in real time.
  2. set up email notifications when Treasure Agent instances go down.

Prerequisites

  • Familiarity with Treasure Agent and Fluentd. In the rest of this article, it is assumed that Treasure Agent is running on Ubuntu Precise (Treasure Agent has deb and rpm packages). With some modifications, one should be able to set it up for Fluentd as well.
  • Treasure Data account: If you do not have a Treasure Data account, you can request access here.

Step 1: Configure the monitoring plugin

SSH into the server that is running a Treasure Agent instance to be monitored. Open its configuration file and make the following edits.

First, configure the td_monitor_agent to send monitoring data to Treasure Data periodically:

<source>
    type td_monitor_agent
    apikey YOUR_TREASURE_DATA_APIKEY
    instance_id monitored_instance_1
</source>

where:

  • apikey
    Your Treasure Data API key. You can find it here.
  • instance_id
    Specify unique instance id across nodes. If not specified, use configuration file path instead.

Next, configure per-plugin monitoring for each output plugin you wish to monitor. To do so, one needs to “wrap” a current output plugin with the td_counter output plugin. For example, suppose you had your Treasure Data output plugin set up as follows:

# BEFORE MONITORING
<match td.*.*>
    type tdlog
    apikey YOUR_API_KEY
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
</match>

Then, in order to monitor this output, configure it as follows:

# AFTER MONITORING
<match td.*.*>
    type td_counter
    <store>
        type tdlog
        apikey YOUR_API_KEY
        auto_create_table
        buffer_type file
        buffer_path /var/log/td-agent/buffer/td
    </store>
</match>

Step 2: Restart Treasure Agent

Restart td-agent to enable the new configuration.

$ sudo service td-agent restart

You should see in Treasure Agent’s log that the two plugins are registered successfully:

# sudo egrep type=\"td_ --color /var/log/td-agent/td-agent.log
2014-05-24 08:47:54 +0000 [info]: adding source type="td_monitor_agent"
2014-05-24 08:47:55 +0000 [info]: adding match pattern="td.*.*" type="td_counter"

Step 3: Send Data and Visualize

We are ready to see the monitoring service in action. Go to the monitoring section of the Treasure Data Console (please log in if you have not done so), and you should see the instance named “monitored_instance_1” (boxed in red). Metrics data is updated every minute.

Treasure Agent monitoring screenshot, Instances list

One can use the drop-down menus (boxed in green) to view both system-wide metrics as well as per-plugin statistics.

Treasure Agent monitoring screenshot, Options

Finally, one can set up email notifications for Treasure Agent availability (boxed in orange). If the email notification checkbox is checked, Treasure Data notifies the user via email if one or more instances have not pinged the monitoring service for more than five minutes.

Treasure Agent monitoring screenshot, Notifications

Resources


Last modified: Oct 10 2016 17:43:30 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.