Skip to content
Products
/
/
/
TD Agent for Storing Logs...
Last updated

TD Agent for Storing Logs Locally and Remotely

You can use Treasure Agent (td-agent’s) ‘copy’ function to store your logs locally as td-agent uploads the data into the cloud.

Prerequisites

  • Basic knowledge of Treasure Data.

  • Basic knowledge of our log collector, td-agent.

Local Disk and Treasure Data

The following configuration shows how to store logs in your local disk (/mnt/archive/logs) and in Treasure Data. The local files are created on an hourly basis.

<match td.*.*>
  type copy
  <store>
    type file
    path /mnt/archive/logs
    time_slice_format %Y%m%d%h
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding file output options, see Fluentd Output Options.

Local MongoDB and Treasure Data

The following configuration shows how to store logs in your local MongoDB ReplicaSet and in Treasure Data. The logs are flushed into MongoDB every 60 seconds.

<match td.*.*>
  type copy
  <store>
    type mongo_replset
    database td
    collection logs
    nodes localhost:27017,localhost:27018,localhost:27019
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding MongoDB output options, see Fluentd out_mongo. There are plugins for various features such as authentication, capped collection, and so on.

Remote Amazon S3 and Treasure Data

The following configuration shows how to store logs in both your Amazon S3 bucket and in Treasure Data. The logs are flushed into Amazon S3 at the top of each hour, meaning that the first upload won’t start immediately. Wait until the top of the next hour to verify the initial test.

<match td.*.*>
  type copy
  <store>
    type s3

    aws_key_id YOUR_AWS_KEY_ID
    aws_sec_key YOUR_AWS_SECRET/KEY
    s3_bucket YOUR_S3_BUCKET_NAME
    path logs/
    buffer_path /var/log/td-agent/buffer/s3

    time_slice_format %Y%m%d-%H
    time_slice_wait 10m
    utc
  </store>
  <store>
    type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    use_ssl true
  </store>
</match>

For more information regarding S3 output options, see Fluentd S3.

Previous page
Next page