Treasure Data provides a server-side agent using Fluentd to collect server-side logs and events. You can continuously import data using Fluentd with the fluent-package.
td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.
- Basic knowledge of Treasure Data.
- Use fluent-package 6 LTS (recommended).
For a quick hands-on experience, see Run a Query and Download Results.
Fluentd is an open source data collector that collects logs from various data sources and uploads them to Treasure Data.
Logs usually rotate on an hourly or daily basis based on time or size. This system quickly produces many large log files that need to be batch imported for further analysis. This is an outdated approach. Logs are better treated as continuously generated STREAMS as opposed to files.
Although bulk-import is supported, we recommend importing your data continuously via Fluentd.