Skip to content
/
Fluentd Continuous Data I...
Last updated

Fluentd Continuous Data Import Integration

You can continuously import the data into Treasure Data using Fluentd.

Fluentd is a daemon written in Ruby that collects logs as JSON streams, buffers the logs, and sends the logs to other systems such as MySQL, MongoDB, or even other instances of Fluentd.

Treasure Agent (td-agent) logging daemon contains Fluentd. However, many users have installed Fluentd via Ruby gems. You can connect your existing Fluentd installations to Treasure Data. Read more about td-agent as a Server Side Agent.

Prerequisites

  • Basic knowledge of Fluentd, and its installation.

Installing the fluent-plugin-td

Install the fluentd and fluent-plugin-td gems as follows:

$ gem install fluentd
$ gem install fluent-plugin-td

Add the following lines to your configuration file. fluent-plugin-td contains the tdlog output type, which is used to upload data into Treasure Data.

# HTTP input
source
  type http
  port 8888
</source>

# TCP input
source
  type forward
  port 24224
</source>

# Treasure Data output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey ...
  auto_create_table
  buffer_type file
  buffer_path /path/to/buffer/td
  use_ssl true
</match>

The API key is a secret key to authenticate your account. You can get your API key from the your profile in TD console.

Confirm Data Upload

Start the Fluentd server.

$ ./bin/fluentd --config fluentd.conf

Add logs in JSON using HTTP.

$ curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/td.testdb.www_access

Fluentd continuously uploads logs, sending data every minute. Sending a SIGUSR1 signal forces Fluentd to flush the buffered logs into the cloud immediately.

$ kill -USR1 pid_of_fluentd

Issue the td tables command to confirm that your data imported successfully.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+

If you run into any issues, Fluentd’s log is a good place to start troubleshooting.

Previous page
Next page