Storing Logs Locally and Remotely

You may want to store your logs locally as td-agent uploads the data into the cloud. In this case, please use td-agent’s ‘copy’ function.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data (an understanding of the Getting Started guide).
  • Basic knowledge of our log collector, td-agent.

Local Disk and Treasure Data

The configuration below will store logs in your local disk (/mnt/archive/logs) and in Treasure Data. The local files are created on an hourly basis.

<match td.*.*>
  type copy
  <store>
    type file
    path /mnt/archive/logs
    time_slice_format %Y%m%d%h
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding file output options, please refer to this page.

Local MongoDB and Treasure Data

The configuration below will store logs in your local MongoDB ReplicaSet and in Treasure Data. The logs are flushed into MongoDB every 60 seconds.

<match td.*.*>
  type copy
  <store>
    type mongo_replset
    database td
    collection logs
    nodes localhost:27017,localhost:27018,localhost:27019
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding MongoDB output options, please refer to this page. There are plugins for various features such as authentication, capped collection, etc.

Remote Amazon S3 and Treasure Data

The configuration below will store logs in both your Amazon S3 bucket and in Treasure Data. The logs are flushed into Amazon S3 at the top of each hour, meaning that the first upload won’t start immediately. Please wait until the top of the next hour to verify the initial test.

<match td.*.*>
  type copy
  <store>
    type s3

    aws_key_id YOUR_AWS_KEY_ID
    aws_sec_key YOUR_AWS_SECRET/KEY
    s3_bucket YOUR_S3_BUCKET_NAME
    path logs/
    buffer_path /var/log/td-agent/buffer/s3

    time_slice_format %Y%m%d-%H
    time_slice_wait 10m
    utc
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding S3 output options, please refer to this page.


Last modified: Oct 10 2014 06:12:21 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.