You can import CSV, TSV, and LTSV formatted logs from Treasure Agent (td-agent), to continuously import the access logs into the cloud.
td-agent handles log-rotation. td-agent keeps a record of the last position of the log, ensuring that each line is read exactly once even if the td-agent process goes down. However, because the information is kept in a file, the "exactly once" guarantee breaks down if the file becomes corrupted.
td-agent is under the Fluentd project. td-agent extends Fluentd with custom plugins for Treasure Data.
Install td-agent on your application servers. td-agent is a daemon program dedicated to the streaming upload of any kind of the time-series data. td-agent is under the Fluentd project. td-agent extends Fluentd with custom plugins for Treasure Data.
To set up td-agent, refer to the following articles; we provide deb/rpm packages for Linux systems.
If you have...
RHEL / CentOS System
AWS Elastic Beanstalk
Specify your authentication key by setting the
apikey option. You can view your API key from the TD Console.
/etc/td-agent/td-agent.conf to set the
YOUR_API_KEY should be your API key string.
Restart your agent when the following lines are in place.
td-agent tails the file, buffers the log (var/log/td-agent/buffer/td), and automatically uploads the log into the cloud.
Confirming Data Import
Sending a SIGUSR1 signal flushes td-agent’s buffer; upload starts immediately.
To confirm that your data uploads successfully, issue the
td tables command as follows.
/var/log/td-agent.log if it’s not working correctly.
td-agent:td-agent must have permission to read the logs.
We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.