Continuous Data Import from Fluentd

This article explains how to continuously import the data into Treasure Data using Fluentd.

Table of Contents

Prerequisites

  • Basic knowledge of Fluentd, and its installation.

About Fluentd

Fluentd is a log collector daemon written in Ruby. It collects logs as JSON streams, buffers them, and sends them to other systems such as MySQL, MongoDB, or even other instances of Fluentd.

Treasure Data’s td-agent logging daemon contains Fluentd. However, many users have installed Fluentd via Ruby gems. This article is written for users who wish to connect their existing Fluentd installations to Treasure Data. Instructions for td-agent (deb/rpm packages) can be found here.

Using fluent-plugin-td

First, please install the fluentd and fluent-plugin-td gems as shown below:

$ gem install fluentd
$ gem install fluent-plugin-td

Next, please add the following lines to your configuration file. fluent-plugin-td contains the tdlog output type, which is used to upload data into Treasure Data.

# HTTP input
<source>
  type http
  port 8888
</source>

# TCP input
<source>
  type forward
  port 24224
</source>

# Treasure Data output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey ...
  auto_create_table
  buffer_type file
  buffer_path /path/to/buffer/td
  use_ssl true
</match>

The apikey is a secret key to authenticate your account. You can get your apikey from the console.

Confirm Data Upload

First, please start the Fluentd server.

$ ./bin/fluentd --config fluentd.conf

Next, add logs in JSON using HTTP.

$ curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/td.testdb.www_access

Fluentd continuously uploads logs, sending data every minute. Sending a SIGUSR1 signal will force fluentd to flush the buffered logs into the cloud immediately.

$ kill -USR1 <pid_of_fluentd>

Issue the td tables command to confirm that your data was imported successfully.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+

If you run into any issues, fluentd’s log is a good place to start investigating.


Last modified: Aug 03 2015 00:01:48 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.