Skip to content
Products
/
/
/
Fluentd Collecting Logs T...
Last updated

Fluentd Collecting Logs Through an HTTP Protocol

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.

You can use HTTP to add logs from many places to Fluentd.

Prerequisites

  • Basic knowledge of Treasure Data, including the TD Toolbelt.

Configuration

Add the following lines to your /etc/fluent/fluentd.conf file.

# HTTP input
<source>
  @type http
  port 8888
</source>

# Treasure Data output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>

Uploading Data via HTTP

You can add logs via HTTP by sending JSON data. Use the POST method and specify the database and table name by URL.

$ curl -X POST -d 'json={"action":"login","user":2}' \
http://localhost:8888/td.testdb.www_access

Fluentd continuously uploads logs every minute. Sending a SIGUSR1 signal flushes the buffer. Uploads start immediately.

kill -USR1 $(cat /var/run/fluent/fluentd.pid)

To confirm that your data has been uploaded successfully, issue the td tables command as shown below.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+

If you run into an issue, your log (/var/log/fluent/fluentd.log) is a good place to start troubleshooting.

Previous page
Next page