Treasure Data provides td-agent to collect server-side logs and events and to seamlessly import the data from Ruby applications.
Basic knowledge of Ruby, Gems, and Bundler.
Basic knowledge of Treasure Data.
Ruby 1.9 or higher (for local testing).
td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.
The td-logger-ruby library enables Ruby applications to post records to their local td-agent. td-agent, in turn, receives the records, buffers them, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.
td-agent Install Options
td-agent, run one of the following commands based on your environment. The agent program is installed automatically by using the package management software for each platform like rpm/deb/dmg.
Ubuntu and Debian
Legacy support for EOL versions is still available
You can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on Amazon Linux.
MacOS X 10.11+
MacOS X 10.11.1 (El Capitan) introduces some security changes. After the td-agent is installed, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.
Windows Server 2012+
The Windows installation requires the steps detailed in:
Opscode Chef Repository
AWS Elastic Beanstalk is also supported. Windows is not supported.
Specify your API key by setting the
apikey option in your
YOUR_API_KEY should be your actual apikey string. You can retrieve your API key from your profile in TD Console. Using the [write-only key](access-control#rest-apis-access) is recommended.
Restart your agent after the following lines are in place.
td-agent accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically uploads the data into the cloud.
Add the ‘td’ gem to your Gemfile.
Initialize and post the records.
Confirming Data Import
Execute the program.
Sending a SIGUSR1 signal flushes td-agent’s buffer. The upload starts immediately.
Using TD Console
To confirm that your data has been uploaded successfully, check your data set.
Or, issue the td tables command if you have a CLI for TD toolbelt.
Use Rack-Based Server Deployments
We recommend that you use unicorn, thin, mongrel, etc. Other setups have not been fully validated.
High Availability Configurations of td-agent
For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.
Monitoring td-agent itself is also important. For general monitoring methods for td-agent, see Monitoring td-agent.
td-agent is fully open-sourced under the Fluentd project.