Storing Logs via Scribe Protocol

td-agent understands the Scribe protocol (Thrift-based). It can also co-exist or replace your existing Scribe infrastructure.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data, including the toolbelt.
  • Basic knowledge of td-agent.

Using Scribe input

The configuration below enables the Scribe input plugin:

<source>
  type scribe
  port 1463
</source>

The category field of the Scribe protocol is used as a tag for td-agent. If you want to store your data in the www_access table within the test_db database, please send your LogEntry as shown below (Ruby example).

entry = LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = 'abcde'
client.Log([entry])

This message is organized as shown below.

{
  message => 'abcde'
}

Message as JSON string

The message field of semi-structured data often contains JSON strings. td-agent uses the msg_format option to parse the JSON string in-place.

<source>
  type scribe
  port 1463
  msg_format json
</source>

Next, please send records as shown below. Please note that entry.message is a JSON string.

entry = LogEntry.new
entry.category = 'td.test_db.www_access'
entry.message = {'a' => 'b', 'c' => d}.to_json
client.Log([entry])

This message is organized as shown below.

{
  'a' => 'b',
  'c' => 'd',
}

Other Formats

The msg_format option supports text, json, and url_param. The format for url_param is as follows.

key1=val1&key2=val2&key3=val3

Further Reading

For more information regarding the Scribe input plugin, please refer to the document below. The plugin also supports Scribe output.


Last modified: Feb 24 2017 09:27:52 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.