TD Agent Using Scribe Protocol to Store Logs

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

Treasure Agent (td-agent) understands the Scribe protocol (Thrift-based). td-agent can co-exist or replace your existing Scribe infrastructure.

See fluent-plugin-scribe. The plugin also supports Scribe output.

• Prerequisites
• Specifying Use of Scribe Input
• Messages as JSON String
• Other Formats

Prerequisites

• Basic knowledge of Treasure Data, including the TD Toolbelt.

• Basic knowledge of td-agent.

Specifying Use of Scribe Input

The following configuration enables the Scribe input plugin:

<source>
  type scribe
  port 1463
</source>  

The category field of the Scribe protocol is used as a tag for td-agent. If you want to store your data in the www_access table within the test_db database, send your LogEntry as follows (Ruby example).

LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = 'abcde'
client.Log([entry])  

This message is organized as follows.

{
  message => 'abcde'
}

Messages as JSON String

The message field of semi-structured data often contains JSON strings. td-agent uses the msg_format option to parse the JSON string in-place.

<source>
  type scribe
  port 1463
  msg_format json
</source>  

Send records as follows. Note that entry.message is a JSON string.

entry = LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = {'a' => 'b', 'c' => d}.to_json
client.Log([entry])  

This message is organized as follows.

{
  'a' => 'b',
  'c' => 'd',
}  

Other Formats

The msg_format option supports text, json, and url_param. The format for url_param is as follows.

key1=val1&key2=val2&key3=val3