Treasure Data provides td-agent to collect server-side logs and events and to seamlessly import the data from Node.js applications.
The fluent-logger-node library enables Node.js applications to post records to their local td-agent.
td-agent, in turn, uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.
Continue to the following topics:
Basic knowledge of Node.js and NPM.
Basic knowledge of Treasure Data, including the TD Toolbelt.
Node.js 0.6 or higher (for local testing).
td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.
The fluent-logger-node library enables Node.js applications to post records to their local td-agent. td-agent, in turn, uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.
td-agent Install Options
td-agent, run one of the following commands based on your environment. The agent program is installed automatically by using the package management software for each platform like rpm/deb/dmg.
Ubuntu and Debian
Legacy support for EOL versions is still available
You can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on Amazon Linux.
MacOS X 10.11+
MacOS X 10.11.1 (El Capitan) introduced some security changes. After the td-agent is installed, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.
Windows Server 2012+
The Windows installation requires the steps detailed in:
Opscode Chef Repository
AWS Elastic Beanstalk is also supported. Windows is currently NOT supported.
Specify your API key by setting the
apikey option. You can view your API key from your profile in TD Console. Set the
apikeyoption in your td-agent.conf file.
YOUR_API_KEY should be your actual API key string. Using a [write-only API key](access-control#rest-apis-access) is recommended.
Restart your agent when the following lines are in place.
td-agent now accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically uploads the data into the cloud.
Obtaining the Most Recent Version
Obtain the most recent version of fluent-logger-node.
A Sample Application
A sample Express app using fluent-logger-node is as follows:
Now use npm to install your dependencies locally:
This is the simplest web app.
Confirming Data Import
Execute the app and go to
http://localhost:3000/ in your browser.
A SIGUSR1 signal is sent and flushes td-agent’s buffer. Upload starts immediately.
To confirm that your data has been uploaded successfully, issue the td tables command as follows:
The first argument of post() determines the database name and table name. If you specify `td.test_db.test_table`, the data is imported into the table *test_table* within the database *test_db*. They are automatically created at upload time.
High-Availability Configurations of td-agent
For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.
Monitoring td-agent itself is also important. Refer to the following document for general monitoring methods for td-agent.
td-agent is fully open-sourced under the Fluentd project.
We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive and Presto Query Languages.