You can continuously import the data into Treasure Data using Fluentd.
Fluentd is a daemon written in Ruby that collects logs as JSON streams, buffers the logs, and sends the logs to other systems such as MySQL, MongoDB, or even other instances of Fluentd.
Treasure Agent (td-agent) logging daemon contains Fluentd. However, many users have installed Fluentd via Ruby gems. You can connect your existing Fluentd installations to Treasure Data. Read more about td-agent as a Server Side Agent.
Basic knowledge of Fluentd, and its installation.
Installing the fluent-plugin-td
Install the fluentd and fluent-plugin-td gems as follows:
Add the following lines to your configuration file. fluent-plugin-td contains the tdlog output type, which is used to upload data into Treasure Data.
The API key is a secret key to authenticate your account. You can get your API key from the your profile in TD console.
Confirm Data Upload
Start the Fluentd server.
Add logs in JSON using HTTP.
Fluentd continuously uploads logs, sending data every minute. Sending a SIGUSR1 signal forces Fluentd to flush the buffered logs into the cloud immediately.
Issue the td tables command to confirm that your data imported successfully.
If you run into any issues, Fluentd’s log is a good place to start troubleshooting.