Fluentd for Storing Logs Locally and Remotely

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

You can use Fluentd's 'copy' function to store your logs locally as Fluentd uploads the data into the cloud.

• Prerequisites
• Local Disk and Treasure Data
• Local MongoDB and Treasure Data
• Remote Amazon S3 and Treasure Data

Prerequisites

• Basic knowledge of Treasure Data.
• Basic knowledge of Fluentd.

Local Disk and Treasure Data

The following configuration shows how to store logs in your local disk (/mnt/archive/logs) and in Treasure Data. The local files are created on an hourly basis.

<match td.*.*>
  @type copy
  <store>
    @type file
    path /mnt/archive/logs
    time_slice_format %Y%m%d%h
  </store>
  <store>
    @type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    use_ssl true
    <buffer>
      @type file
      path /var/log/fluent/buffer/td
    </buffer>
  </store>
</match>

For more information regarding file output options, see Fluentd Output Options.

Local MongoDB and Treasure Data

The following configuration shows how to store logs in your local MongoDB ReplicaSet and in Treasure Data. The logs are flushed into MongoDB every 60 seconds.

<match td.*.*>
  @type copy
  <store>
    @type mongo_replset
    database td
    collection logs
    nodes localhost:27017,localhost:27018,localhost:27019
  </store>
  <store>
    @type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    use_ssl true
    <buffer>
      @type file
      path /var/log/fluent/buffer/td
    </buffer>
  </store>
</match>

For more information regarding MongoDB output options, see Fluentd out_mongo. There are plugins for various features such as authentication, capped collection, and so on.

Remote Amazon S3 and Treasure Data

The following configuration shows how to store logs in both your Amazon S3 bucket and in Treasure Data. The logs are flushed into Amazon S3 at the top of each hour, meaning that the first upload won't start immediately. Wait until the top of the next hour to verify the initial test.

<match td.*.*>
  @type copy
  <store>
    @type s3

    aws_key_id YOUR_AWS_KEY_ID
    aws_sec_key YOUR_AWS_SECRET/KEY
    s3_bucket YOUR_S3_BUCKET_NAME
    path logs/

    time_slice_format %Y%m%d-%H
    time_slice_wait 10m
    utc

    <buffer>
      @type file
      path /var/log/fluent/buffer/s3
    </buffer>
  </store>
  <store>
    @type tdlog
    endpoint api.treasuredata.com
    apikey YOUR_API_KEY_HERE
    auto_create_table
    use_ssl true
    <buffer>
      @type file
      path /var/log/fluent/buffer/td
    </buffer>
  </store>
</match>

For more information regarding S3 output options, see Fluentd S3.