Skip to content
Products
/
/
/
Fluentd Using Scribe Prot...
Last updated

Fluentd Using Scribe Protocol to Store Logs

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.

Fluentd understands the Scribe protocol (Thrift-based). Fluentd can co-exist or replace your existing Scribe infrastructure.

See fluent-plugin-scribe. The plugin also supports Scribe output.

Prerequisites

  • Basic knowledge of Treasure Data, including the TD Toolbelt.
  • Basic knowledge of Fluentd.

Specifying Use of Scribe Input

The following configuration enables the Scribe input plugin:

<source>
  @type scribe
  port 1463
</source>

The category field of the Scribe protocol is used as a tag for Fluentd. If you want to store your data in the www_access table within the test_db database, send your LogEntry as follows (Ruby example).

LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = 'abcde'
client.Log([entry])

This message is organized as follows.

{
  message => 'abcde'
}

Messages as JSON String

The message field of semi-structured data often contains JSON strings. Fluentd uses the msg_format option to parse the JSON string in-place.

<source>
  @type scribe
  port 1463
  msg_format json
</source>

Send records as follows. Note that entry.message is a JSON string.

entry = LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = {'a' => 'b', 'c' => d}.to_json
client.Log([entry])

This message is organized as follows.

{
  'a' => 'b',
  'c' => 'd',
}

Other Formats

The msg_format option supports text, json, and url_param. The format for url_param is as follows.

key1=val1&key2=val2&key3=val3

Previous page
Next page