Skip to content
Tools
/
/
Fluentd Continuous Data I...
Last updated

Fluentd Continuous Data Import Integration

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.

You can continuously import the data into Treasure Data using Fluentd.

Fluentd is a daemon written in Ruby that collects logs as JSON streams, buffers the logs, and sends the logs to other systems such as MySQL, MongoDB, or even other instances of Fluentd.

Prerequisites

  • Basic knowledge of Fluentd, and its installation.

Installing Fluentd

Fluentd (fluent-package) Install Options

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation. For migration guidance from td-agent, see Fluentd Installation Guide.

To install fluent-package, run one of the following commands based on your environment.

RHEL/CentOS/Rocky Linux

# fluent-package 6 LTS (recommended)
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-redhat-fluent-package6-lts.sh | sh

Ubuntu

# Ubuntu 24.04 Noble - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package6-lts.sh | sh

# Ubuntu 22.04 Jammy - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package6-lts.sh | sh

Debian

# Debian Bookworm - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-debian-bookworm-fluent-package6-lts.sh | sh

Amazon Linux

# Amazon Linux 2023 - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-amazon2023-fluent-package6-lts.sh | sh

Windows

Download the MSI installer from:

After installation:

  1. Edit the configuration file at C:/opt/fluent/etc/fluent/fluentd.conf
  2. Start the service using net start fluentdwinsvc or via Services administrative tool

macOS

fluent-package for macOS is planned to be available via Homebrew. For current installation options, see Fluentd Installation Guide.

Starting the Service

After installation, start and verify the Fluentd service.

Linux

sudo systemctl start fluentd.service
sudo systemctl status fluentd.service

The configuration file is located at /etc/fluent/fluentd.conf.

Windows

net start fluentdwinsvc

The configuration file is located at C:\opt\fluent\etc\fluent\fluentd.conf.

macOS (gem installation)

fluentd -c /path/to/fluentd.conf

For more details, see the Fluentd Documentation.

Installing the fluent-plugin-td

If you're using Fluentd installed via Ruby gems instead of fluent-package, install the fluentd and fluent-plugin-td gems as follows:

gem install fluentd
gem install fluent-plugin-td

Configuration

Add the following lines to your configuration file (/etc/fluent/fluentd.conf for fluent-package). The fluent-plugin-td contains the tdlog output type, which is used to upload data into Treasure Data.

# HTTP input
<source>
  @type http
  port 8888
</source>

# TCP input
<source>
  @type forward
  port 24224
</source>

# Treasure Data output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>

The API key is a secret key to authenticate your account. You can get your API key from your profile in TD console.

Confirm Data Upload

Start the Fluentd service (if using fluent-package):

sudo systemctl start fluentd.service

Or start Fluentd manually (if using gem installation):

fluentd --config fluentd.conf

Add logs in JSON using HTTP.

curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/td.testdb.www_access

Fluentd continuously uploads logs, sending data every minute. Sending a SIGUSR1 signal forces Fluentd to flush the buffered logs into the cloud immediately.

kill -USR1 $(cat /var/run/fluent/fluentd.pid)

Issue the td tables command to confirm that your data imported successfully.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+

If you run into any issues, Fluentd's log (/var/log/fluent/fluentd.log) is a good place to start troubleshooting.

Other Resources

Previous page
Next page