Treasure Agent Monitoring Service: Quickstart
Table of Contents
This article shows how to set up the monitoring service for Treasure Agent. The monitoring service provides the ability to:
- visualize and monitor Treasure Agent’s resource usage with a per-plugin level granularity in real time.
- set up email notifications when Treasure Agent instances go down.
- Familiarity with Treasure Agent and Fluentd. In the rest of this article, it is assumed that Treasure Agent is running on Ubuntu Precise (Treasure Agent has deb and rpm packages). With some modifications, one should be able to set it up for Fluentd as well.
- Treasure Data account: If you do not have a Treasure Data account, you can request access here.
Step 1: Configure the monitoring plugin
SSH into the server that is running a Treasure Agent instance to be monitored. Open its configuration file and make the following edits.
First, configure the td_monitor_agent to send monitoring data to Treasure Data periodically:
<source> type td_monitor_agent apikey YOUR_TREASURE_DATA_APIKEY instance_id monitored_instance_1 </source>
Your Treasure Data API key. You can find it here.
Specify unique instance id across nodes. If not specified, use configuration file path instead.
Next, configure per-plugin monitoring for each output plugin you wish to monitor. To do so, one needs to “wrap” a current output plugin with the td_counter output plugin. For example, suppose you had your Treasure Data output plugin set up as follows:
# BEFORE MONITORING <match td.*.*> type tdlog apikey YOUR_API_KEY auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td </match>
Then, in order to monitor this output, configure it as follows:
# AFTER MONITORING <match td.*.*> type td_counter <store> type tdlog apikey YOUR_API_KEY auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td </store> </match>
Step 2: Restart Treasure Agent
Restart td-agent to enable the new configuration.
$ sudo service td-agent restart
You should see in Treasure Agent’s log that the two plugins are registered successfully:
# sudo egrep type=\"td_ --color /var/log/td-agent/td-agent.log 2014-05-24 08:47:54 +0000 [info]: adding source type="td_monitor_agent" 2014-05-24 08:47:55 +0000 [info]: adding match pattern="td.*.*" type="td_counter"
Step 3: Send Data and Visualize
We are ready to see the monitoring service in action. Go to the monitoring section of the Treasure Data Console (please log in if you have not done so), and you should see the instance named “monitored_instance_1” (boxed in red). Metrics data is updated every minute.
One can use the drop-down menus (boxed in green) to view both system-wide metrics as well as per-plugin statistics.
Finally, one can set up email notifications for Treasure Agent availability (boxed in orange). If the email notification checkbox is checked, Treasure Data notifies the user via email if one or more instances have not pinged the monitoring service for more than five minutes.
Last modified: Oct 10 2016 17:43:30 UTC