You can send data from Kafka to Treasure Data, using the Fluentd consumer for Kafka. Other than the consumer itself, and depending on your current setup, there may be a few additional requirements.
This document assumes Ubuntu Precise Debian; commands should otherwise be chosen according to the operating system.
This feature is in Beta. For more information, contact firstname.lastname@example.org.
Basic knowledge of Treasure Data
Basic knowledge of Apache Kafka setup, configuration, and architecture including producers and consumers
Ability to build with Gradle
Working knowledge of Linux
Working knowledge of Fluentd
A working Kafka single-node or multi-broker installation
Install and configure Fluentd
Review the pre-setup and installation instructions for installing Fluentd. Ensure the following before setting up fluentd:
Installed ntp server and set up proper restrict values in /etc/ntp.conf
Added the local clock as NTP backup and set also logging parameters
Restarted the NTP service daemon
Configured the correct number of file descriptors in /etc/security/limits.conf
Refer to Treasure Agent (td-agent) installation for other distributions.
You will be using the Fluentd Consumer for Kafka to forward your events (fluentd receives events from default port of 24224) so your configuration file should look similar to the following:
Set your Treasure Data Write-Only API as an environment variable on your machine. Then, you might want to restart Fluentd:
Installing and Configuring Kafka Consumer for Fluentd
Download the JAR File
Download pre-compiled JAR file from the GitHub releases page.
The jar file name is
Download Configuration Files
You also must download two configuration files from GitHub.
fluentd-consumer.properties: Use to set Kafka consumer and log forwarding parameters
log4j.properties: Use to set logging parameters of consumer itself.
Here is file list example of kafka-fluentd-consumer:
Upgrade Java on Host
Make sure you are running at least Java version 1.7:
If not, complete the following steps:
Run Kafka Fluentd Consumer
Create a Test Topic
Send Multiple Messages in JSON Format
The Kafka Fluentd Consumer requires messages to be sent in JSON format.
Confirm the Messages by Starting a Consumer
You should see:
Configure and Run the Kafka Fluentd Consumer
Modify fluentd-consumer.properties with an appropriate configuration. Change the topic name in the following to your intended Treasure Data table name. fluentd.consumer.topics=test.
Run the command to start the consumer. You may need to adjust this command line to account for the paths to the relevant .jar and .properties files.
You can go back and send a few more events from your producer, as long as they are in valid JSON format.
Query your Data on Treasure Data
Run the following query on TD Console:
Under Query Result, you can see if the run was successful.