Skip to content
Tools
/
/
Rails Apps Import
Last updated

Rails Apps Import Integration

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.

Treasure Data provides Fluentd to collect server-side logs and events and to seamlessly import the data from Ruby on Rails applications.

Prerequisites

  • Basic knowledge of Ruby, Rails, Gems, and Bundler.
  • Basic knowledge of Treasure Data.
  • Ruby 3.3 or higher (for local testing).

Installing Fluentd

Install Fluentd (fluent-package) on your application servers. Fluentd sits within your application servers, focusing on uploading application logs to the cloud.

The td-logger-ruby library enables Ruby on Rails applications to post records to their local Fluentd. Fluentd, in turn, receives the records, buffers them, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

Fluentd (fluent-package) Install Options

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation. For migration guidance from td-agent, see Fluentd Installation Guide.

To install fluent-package, run one of the following commands based on your environment.

RHEL/CentOS/Rocky Linux

# fluent-package 6 LTS (recommended)
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-redhat-fluent-package6-lts.sh | sh

Ubuntu

# Ubuntu 24.04 Noble - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package6-lts.sh | sh

# Ubuntu 22.04 Jammy - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package6-lts.sh | sh

Debian

# Debian Bookworm - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-debian-bookworm-fluent-package6-lts.sh | sh

Amazon Linux

# Amazon Linux 2023 - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-amazon2023-fluent-package6-lts.sh | sh

Windows

Download the MSI installer from:

After installation:

  1. Edit the configuration file at C:/opt/fluent/etc/fluent/fluentd.conf
  2. Start the service using net start fluentdwinsvc or via Services administrative tool

macOS

fluent-package for macOS is planned to be available via Homebrew. For current installation options, see Fluentd Installation Guide.

Starting the Service

After installation, start and verify the Fluentd service.

Linux

sudo systemctl start fluentd.service
sudo systemctl status fluentd.service

The configuration file is located at /etc/fluent/fluentd.conf.

Windows

net start fluentdwinsvc

The configuration file is located at C:\opt\fluent\etc\fluent\fluentd.conf.

macOS (gem installation)

fluentd -c /path/to/fluentd.conf

For more details, see the Fluentd Documentation.

Modifying fluentd.conf

Specify your API key by setting the apikey option in your /etc/fluent/fluentd.conf file (for fluent-package).

# Input from Logging Libraries
<source>
  @type forward
  port 24224
</source>

# Treasure Data Output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>

YOUR_API_KEY should be your actual apikey string. You can retrieve your API key from your profiles in TD Console. Using a write-only API key is recommended.

Restart the Fluentd service after adding the following lines:

# Linux
sudo systemctl restart fluentd.service

# macOS (gem installation)
# Restart the Fluentd process manually

Fluentd accepts data via port 24224, buffers the data (/var/log/fluent/buffer/td), and automatically uploads the data into the cloud.

Using td-logger-ruby

Add the ‘td’ gem to your Gemfile.

gem 'td', "~> 0.10.6"

Modify the config/treasure_data.yml file as follows:

production:
  agent: "localhost:24224"
  tag: td.production_database_name
  debug_mode: false

development:
  agent: "localhost:24224"
  tag: td.development_database_name
  debug_mode: false

test:

Insert the appropriate logging code into your application.

# Example1: login event
TD.event.post('login', {:uid=>123})

# Example2: follow event
TD.event.post('follow', {:uid=>123, :from=>'TD', :to=>'Heroku'})

# Example3: pay event
TD.event.post('pay',
              {:uid=>123, :item_name=>'Stone of Jordan',
               :category=>'ring', :price=>100, :count=>1})

Confirming Data Import

Execute the program.

$ ruby test.rb

Sending a SIGUSR1 signal flushes Fluentd's buffer. The upload starts immediately.

# Linux
$ kill -USR1 $(cat /var/run/fluent/fluentd.pid)

# macOS (gem installation)
# Send SIGUSR1 to the Fluentd process

From TD Console

To confirm that your data has been uploaded successfully, check your data set.

From CLI

Or, use the td tables command if you have a CLI.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| test_db    | login      | log  | 1         |
| test_db    | follow     | log  | 1         |
| test_db    | pay        | log  | 1         |
+------------+------------+------+-----------+

Production Deployments

Use Rack-Based Server Deployments

We recommend that you use unicorn, thin, mongrel, etc. Other setups have not been fully validated.

High Availability Configurations of Fluentd

For high-traffic websites (more than 5 application nodes), use a high availability configuration of Fluentd to improve data transfer reliability and query performance.

Monitoring Fluentd

Monitoring Fluentd itself is also important. For general monitoring methods for Fluentd, see Monitoring Fluentd.

Fluentd is fully open-sourced under the Fluentd project.

Previous page
Next page