Tailing JSON Formatted Logs

This article explains how to tail JSON formatted logs from td-agent, to continuously import the access logs into the cloud.

Table of Contents

Installing td-agent

‘td-agent’ needs to be installed on your application servers. td-agent is a daemon program dedicated to the streaming upload of any kind of the time-series data. td-agent is developed and maintained by Treasure Data, Inc.

To set up td-agent, please refer to the following articles; we provide deb/rpm packages for Linux systems.

If you have... Please refer to...
MacOS X Installing td-agent on MacOS X
Ubuntu System Installing td-agent for Debian and Ubuntu
RHEL / CentOS System Installing td-agent for Redhat and CentOS
AWS Elastic Beanstalk Installing td-agent on AWS Elastic Beanstalk
td-agent is fully open-sourced under the fluentd project. td-agent extends fluentd with custom plugins for Treasure Data.

Modifying /etc/td-agent/td-agent.conf

Next, please specify your authentication key by setting the apikey option. You can view your api key from the console. Next, please set the apikey option in your td-agent.conf file.

Note: YOUR_API_KEY should be your actual apikey string.

# Tailing the JSON formatted Logs
  type tail
  format json
  tag td.production.foo
  path /path/to/the/file/foo.json
  pos_file /var/log/td-agent/foo.pos

# Treasure Data Input and Output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true

Please restart your agent once these lines are in place.

$ sudo /etc/init.d/td-agent restart

td-agent will now tail the file, buffer it (var/log/td-agent/buffer/td), and automatically upload it into the cloud.

Confirming Data Import

Here is a sample log file. Every time a new line is appended to the log file, td-agent parses the line and adds it to its buffer. td-agent uploads the data into the cloud every 5 minutes; to upload the data immediately, please send a SIGUSR1 signal.

$ tail -n 5 /path/to/the/file/foo.json
{"a"=>"b", "c"=>"d"}
{"a"=>"b", "c"=>"d", "e"=>1}
{"a"=>"b", "c"=>"d", "e"=>1, "f"=>2.0}
{"a"=>"b", "c"=>"d"}
{"a"=>"b", "c"=>"d", "e"=>1}

Issue the commands below to confirm that everything is configured correctly.

# append new entries
$ tail -n 3 /path/to/the/file/foo.json > sample.txt # take the last three lines of the log...
$ cat sample.txt >>/path/to/the/file/foo.json       # and append them to the buffer file to trigger the tail plugin.

# flush the buffer
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`
td-agent handles log-rotation. td-agent keeps a record of the last position of the log, ensuring that each line is read exactly once even if the td-agent process goes down. However, since the information is kept in a file, the "exactly once" guarantee breaks down if the file becomes corrupted.

To confirm that your data has been uploaded successfully, issue the td tables command as shown below.

$ td tables
| Database   | Table      | Type | Count     |
| production | foo        | log  | 3         |

Please check /var/log/td-agent.log if it’s not working correctly. td-agent:td-agent needs to have a permission to read the logs.

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.

Last modified: May 19 2016 02:28:09 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.