# Fluentd Continuous Data Import Integration

td-agent was discontinued in December 2023 and has been replaced by **fluent-package**. The fluent-package is the official successor maintained by the [Cloud Native Computing Foundation](https://www.cncf.io/projects/).

You can continuously import the data into Treasure Data using Fluentd.

[Fluentd](http://fluentd.org/) is a daemon written in Ruby that collects logs as JSON streams, buffers the logs, and sends the logs to other systems such as MySQL, MongoDB, or even other instances of Fluentd.

## Prerequisites

- Basic knowledge of [Fluentd](http://fluentd.org/), and its installation.


## Installing Fluentd

## Fluentd (fluent-package) Install Options

td-agent was discontinued in December 2023 and has been replaced by **fluent-package**. The fluent-package is the official successor maintained by the [Cloud Native Computing Foundation](https://www.cncf.io/projects/). For migration guidance from td-agent, see [Fluentd Installation Guide](https://docs.fluentd.org/installation/install-fluent-package).

To install `fluent-package`, run one of the following commands based on your environment.

### RHEL/CentOS/Rocky Linux


```bash
# fluent-package 6 LTS (recommended)
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-redhat-fluent-package6-lts.sh | sh
```

### Ubuntu


```bash
# Ubuntu 24.04 Noble - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package6-lts.sh | sh

# Ubuntu 22.04 Jammy - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package6-lts.sh | sh
```

### Debian


```bash
# Debian Bookworm - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-debian-bookworm-fluent-package6-lts.sh | sh
```

### Amazon Linux


```bash
# Amazon Linux 2023 - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-amazon2023-fluent-package6-lts.sh | sh
```

### Windows

Download the MSI installer from:

- [fluent-package 6 LTS for Windows](https://fluentd.cdn.cncf.io/lts/6/windows/index.html)


After installation:

1. Edit the configuration file at `C:/opt/fluent/etc/fluent/fluentd.conf`
2. Start the service using `net start fluentdwinsvc` or via Services administrative tool


### macOS

fluent-package for macOS is planned to be available via Homebrew. For current installation options, see [Fluentd Installation Guide](https://docs.fluentd.org/installation).

## Starting the Service

After installation, start and verify the Fluentd service.

### Linux


```bash
sudo systemctl start fluentd.service
sudo systemctl status fluentd.service
```

The configuration file is located at `/etc/fluent/fluentd.conf`.

### Windows


```cmd
net start fluentdwinsvc
```

The configuration file is located at `C:\opt\fluent\etc\fluent\fluentd.conf`.

### macOS (gem installation)


```bash
fluentd -c /path/to/fluentd.conf
```

For more details, see the [Fluentd Documentation](https://docs.fluentd.org/).

## Installing the fluent-plugin-td

If you're using Fluentd installed via Ruby gems instead of fluent-package, install the fluentd and fluent-plugin-td gems as follows:


```bash
gem install fluentd
gem install fluent-plugin-td
```

## Configuration

Add the following lines to your configuration file (`/etc/fluent/fluentd.conf` for fluent-package). The fluent-plugin-td contains the tdlog output type, which is used to upload data into Treasure Data.


```conf
# HTTP input
<source>
  @type http
  port 8888
</source>

# TCP input
<source>
  @type forward
  port 24224
</source>

# Treasure Data output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>
```

The API key is a secret key to authenticate your account. You can get your API key from your profile in TD console.

## Confirm Data Upload

Start the Fluentd service (if using fluent-package):


```bash
sudo systemctl start fluentd.service
```

Or start Fluentd manually (if using gem installation):


```bash
fluentd --config fluentd.conf
```

Add logs in JSON using HTTP.


```bash
curl -X POST -d 'json={"action":"login","user":2}' \
  http://localhost:8888/td.testdb.www_access
```

Fluentd continuously uploads logs, sending data every minute. Sending a SIGUSR1 signal forces Fluentd to flush the buffered logs into the cloud immediately.


```bash
kill -USR1 $(cat /var/run/fluent/fluentd.pid)
```

Issue the `td tables` command to confirm that your data imported successfully.


```bash
$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+
```

If you run into any issues, Fluentd's log (`/var/log/fluent/fluentd.log`) is a good place to start troubleshooting.

## Other Resources

- [Fluentd Documentation](https://docs.fluentd.org/)