Skip to content
/
Perl Apps Import Integrat...
Last updated

Perl Apps Import Integration

Treasure Data provides td-agent to collect server-side logs and events and to seamlessly import the data from Perl applications.

Prerequisites

  • Basic knowledge of Perl and CPAN.
  • Basic knowledge of Treasure Data, including the latest installed version of the TD Toolbelt.
  • Perl 5.10 or higher (for local testing).

Installing td-agent

Install td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.

The fluent-logger-perl library enables Perl applications to post records to their local td-agent. td-agent, in turn, uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

td-agent Install Options

To install td-agent, run one of the following commands based on your environment. The agent program is installed automatically when you use the package management software for each platform like rpm/deb/dmg.

RHEL/CentOS 5,6,7

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

Ubuntu and Debian

# 18.04 Bionic
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
# 16.04 Xenial (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh

Amazon Linux

You can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on Amazon Linux.

MacOS X 10.11+

$ open 'https://td-agent-package-browser.herokuapp.com/3/macosx/td-agent-3.1.1-0.dmg'

MacOS X 10.11.1 (El Capitan) introduces some security changes. After the td-agent is installed, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.

Windows Server 2012+

The Windows installation requires the steps detailed in:

Opscode Chef Repository

$ echo 'cookbook "td-agent"' >> Berksfile
$ berks install

AWS Elastic Beanstalk is also supported. Windows is currently NOT supported.

Modifying /etc/td-agent/td-agent.conf

Specify your API key by setting thee your API key by setting the apikey option. You can view your API key from your profiles in TD Console. Set the apikey option in your td-agent.conf file.

# Treasure Data Input and Output
source
  type forward
  port 24224
</source>
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>

YOUR_API_KEY should be your actual apikey string. You can retrieve your API key from your profiles in TD Console. Using a write-only API key is recommended.

Restart your agent when the following lines are in place.

# Linux
$ sudo /etc/init.d/td-agent restart

# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

td-agent now accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically upload the data into the cloud.

Using fluent-logger-perl

First, install the Fluent::Logger library via CPAN.

$ cpan
cpan[1]> install Fluent::Logger

Next, initialize and post the records as follows.

test.pl

use Fluent::Logger;
my $logger = Fluent::Logger->new(
    host => '127.0.0.1',
    port => 24224,
    tag_prefix => 'td.test_db',
);
$logger->post("test_table", { "entry1" => "value1", "entry2" => 2 });

Confirming Data Import

First, execute the preceding program.

$ perl test.pl

Sending a SIGUSR1 signal flushed td-agent’s buffer. Upload starts immediately.

# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`

# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`

To confirm that your data has been uploaded successfully, issue the td tables command as follows:

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| test_db    | test_table | log  | 1         |
+------------+------------+------+-----------+

The first argument of post() determines the database name and table name. If you specify td.test\_db.test\_table, the data is imported into the table *test_table* within the database *test_db*. They are automatically created at upload time.

Production Deployments

High-Availability Configurations of td-agent

For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.

Monitoring td-agent

Monitoring td-agent itself is also important. For general monitoring methods for td-agent see the following:

td-agent is fully open-sourced under the Fluentd project.

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive and Presto Query Languages.

Previous page
Next page