TD Agent Collecting Logs Through an Http Protocol

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

You can use HTTP to add logs from many places to Treasure Agent (td-agent).

• Prerequisites
• Configuration
• Uploading Data via HTTP

Prerequisites

• Basic knowledge of Treasure Data, including the TD Toolbelt.

Configuration

Add the following lines to your /etc/td-agent/td-agent.conf file.

# HTTP input
<source>
  type http
  port 8888
</source>

# Treasure Data output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey ...
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>

Uploading Data via HTTP

You can add logs via HTTP by sending JSON data. Use the POST method and specify the database and table name by URL.

$ curl -X POST -d 'json={"action":"login","user":2}' \
http://localhost:8888/td.testdb.www_access

td-agent continuously uploads logs every minute. Sending a SIGUSR1 signal flushes the buffer. Uploads start immediately.

kill -USR1 `cat /var/run/td-agent/td-agent.pid`

To confirm that your data has been uploaded successfully, issue the td tables command as shown below.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| testdb     | www_access | log  | 1         |
+------------+------------+------+-----------+

If you run into an issue, your log (/var/log/td-agent.log) is a good place to start troubleshooting.