Treasure Data provides td-agent to collect server-side logs and events and to import the data from Scala applications.
Basic knowledge of Scala.
Basic knowledge of Treasure Data, including the TD Toolbelt.
JVM, Scala, sbt v0.11 or later.
td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.
The td-logger-java library enables Scala applications to post records to their local td-agent. td-agent, in turn, uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.
td-agent Install Options
td-agent, run one of the following commands based on your environment. The agent program is installed automatically by using the package management software for each platform like rpm/deb/dmg.
Ubuntu and Debian
Legacy support for EOL versions is still available
You can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on Amazon Linux.
MacOS X 10.11+
MacOS X 10.11.1 (El Capitan) introduced some security changes. After installing the td-agent, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.
Windows Server 2012+
The Windows installation requires multiple steps. Complete the steps documented:
Opscode Chef (repository)
You can read more about the repository.
AWS Elastic Beanstalk is also supported. Windows is not supported.
Next, specify your API key by setting the
apikey option. You can retrieve your API key from your profile in TD Console. Set the
apikey option in your td-agent.conf file.
YOUR_API_KEY should be your actual apikey string. Using a [write-only API key](access-control#rest-apis-access) is recommended.
Restart your agent when the following lines are in place.
td-agent now accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically uploads the data into the cloud.
First, add the following lines to build.sbt. The logger’s revision information can be found in CHANGES.txt.
If you need an all-in-one jar file, we provide one at http://central.maven.org/maven2/com/treasuredata/.
Next, configure your treasure-data.properties file using the following commands:
Finally, insert the following lines into your application to initialize and post records. You can read more information about the API.
This example expects the following structure.
Confirming Data Import
First, execute the preceding program.
Sending a SIGUSR1 signal flushes td-agent’s buffer. Upload starts immediately.
To confirm the data upload, use
High-Availability Configurations of td-agent
For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.
Monitoring td-agent itself is also important. Refer to the following document for general monitoring methods for td-agent:
td-agent is fully open-sourced under the Fluentd project.
We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive and Presto Query Languages.