Server-Side Agent with Java Apps

Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. This article explains 4 steps to streamingly import the data from Java applications, through Treasure Agent.

Table of Contents

Prerequisites

  • Basic knowledge of Java.
  • Basic knowledge of Maven2.
  • Basic knowledge of Treasure Data.
  • Java 6 or higher (for local testing).

What is Treasure Agent?

First of all, Treasure Agent (td-agent) needs to be installed on your application servers. Treasure Agent is an agent program sits within your application servers, focusing on uploading application logs to the cloud.



The td-logger-java library enables Java applications to post records to their local Treasuge Agent. Treasure Agent in turn receives the records, buffers them, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

How to install Treasure Agent?

This video demonstrates how to install Treasure Agent in 3 minutes.

Step 1: Installing Treasure Agent

To install Treasure Agent (td-agent), please execute one of the command below based on your environment. The agent program will be installed automatically by using the package management software for each platform like rpm/deb/dmg.

RHEL/CentOS 5,6,7

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

Ubuntu & Debian

# 14.04 Trusty (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh
# 10.04 Lucid
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-lucid-td-agent2.sh | sh

# Debian Squeeze (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh
# Debian Wheezy (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-wheezy-td-agent2.sh | sh

Amazon Linux

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

MacOS X 10.11+

$ open 'https://packages.treasuredata.com/2/macosx/td-agent-2.3.0-0.dmg'
Untitled-3
With MacOS X 10.11.1 (El Capitan), some security changes were introduced and we are testing the changes we made to td-agent for this version of OS. For now, once the td-agent is installed, please edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.

Windows Server 2012+

Windows installation needs multiple steps to follow. Please go to this documentation.



Opscode Chef (repository)

$ echo 'cookbook "td-agent"' >> Berksfile
$ berks install

AWS Elastic Beanstalk is also supported. Windows is currently NOT supported.

Step 2: Modifying /etc/td-agent/td-agent.conf

Next, please specify your API key by setting the apikey option in your /etc/td-agent/td-agent.conf file.

# Input from Logger Libraries
<source>
  type forward
  port 24224
</source>

# Treasure Data Output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>
Untitled-3
YOUR_API_KEY should be your actual apikey string. You can retrieve your api key from HERE. Using a [write-only API key](access-control#rest-apis-access) is recommended.

Please restart your agent once these lines are in place.

# Linux
$ sudo /etc/init.d/td-agent restart

# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

td-agent will now accept data via port 24224, buffer it (var/log/td-agent/buffer/td), and automatically upload it into the cloud.

Step 3: Using td-logger-java

Untitled-3
If you need an all-in-one jar file, we provide one at http://central.maven.org/maven2/com/treasuredata/. Be sure to download the latest version of the td-logger-{version_number}-jar-with-dependencies.jar file.

Conversely, if you are using Maven, please add the following lines to pom.xml in your Maven project:

<dependencies>
  ...
  <dependency>
    <groupId>com.treasuredata</groupId>
    <artifactId>td-logger</artifactId>
    <version>${logger.version}</version>
  </dependency>
  ...
</dependencies>

Next, please configure your treasure-data.properties file using the commands shown below:

td.logger.agentmode=true
td.logger.agent.host=localhost
td.logger.agent.port=24224
td.logger.agent.tag=td

You must ensure that the treasure-data.properties file is referenced by your java classpath.

Finally, please insert the following lines into your application. Further information regarding the API can be found here.

import com.treasure_data.logger.TreasureDataLogger;
public class Main {
  private static TreasureDataLogger LOG;
  static {
    try {
      Properties props = System.getProperties();
      props.load(Main.class.getClassLoader().getResourceAsStream("treasure-data.properties"));
      LOG = TreasureDataLogger.getLogger("test_db");
    } catch (IOException e) {
      // do something
    }
  }
  public void doApp() {
    Map<String, Object> data = new HashMap<String, Object>();
    data.put("from", "userA");
    data.put("to", "userB");
    LOG.log("follow", data);
  }
}

Step 4: Confirming Data Import

First, please execute the program above.

# Post a record
$ java -jar test.jar

Sending a SIGUSR1 signal will flush td-agent’s buffer; upload will start immediately.

# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`

# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`

From Web Console

To confirm that your data has been uploaded successfully, check your dataset from the web browser HERE.

From CLI

Or, please issue the td tables command if you have a CLI client.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| test_db    | follow     | log  | 1         |
+------------+------------+------+-----------+

Production Deployments

High-Availablability Configurations of td-agent

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

Monitoring td-agent

Monitoring td-agent itself is also important. Please refer to this document for general monitoring methods for td-agent.

Untitled-3
td-agent is fully open-sourced under the fluentd project.

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.


Last modified: Oct 03 2016 10:50:26 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.