Logstash is an open source software for log management and widely used as a part of the ELK stack.
Many plugins exist for Logstash to collect, filter, and store data from many sources, and into many destinations. You can ingest data from Logstash into Treasure Data by creating a Treasure Data plugin.
Install and Configure the Treasure Data Logstash Plugin
You can install the Treasure Data plugin for Logstash. The following example assumes that you already have Logstash installed and configured.
Configure Logstash with Treasure Data services. You must provide the name of a database and table into which your Logdash data is imported. You can retrieve your API key from your profile in TD Console. Use your write-only key.
Launch Logstash with the configuration file.
You can see rows of your data in TD Console. Log message texts are stored in message column, and some additional columns will exist (for example: time, host and version).
Logs are stored in a table. Create 2 or more sections in your configuration file if you want to insert data into 2 or more tables.
In your Logstash configuration, you can specify the following options:
auto_create_table [true]: creates a table if table doesn’t exists
The plugin work with default values for almost all cases, but some of the parameters might help you in unstable network environments.
This plugin buffers data in memory buffer in 5 minutes at most. Buffered data is lost if the Logstash process crashes. To avoid losing your data during import, we recommend to use td-agent with your Logstash plugin.
The Logstash Plugin in Combination with td-agent
The Logstash plugin is limited in the areas of buffering, stored table specifications, and performance.
You can use Treasure Agent (td-agent) for more flexible and high performance transferring of data. You use logstash-output-fluentd to do it.
Logstash can be configured to send the logs to a td-agent node, and that td-agent stores all the data into Treasure Data.
The Fluentd tdlog plugin can store data into many database-table combinations by parsing td.dbname.tablename. You can configure any database and table pairs in Logstash configuration files.