C/C++ Data Import with Fluent Bit

This article describes how to import the data reliably from your C/C++ applications, using Fluent Bit.

Table of Contents

Prerequisites

  • Basic knowledge of C/C++ Application Development.
  • Basic knowledge of Treasure Data.

What is Fluent Bit?

Fluent Bit is a specialized open-source data collector tool for embedded operating systems and is developed by Treasure Data. Fluent Bit can be used as a standalone tool and can also be embedded in your C or C++ applications.

By emdedding Fluent Bit in your C/C++ applications, you can have robust and light-weight data collection mechanism (buffering, retry, etc) inside your applications.

How to Embed Fluent Bit and connect to Treasure Data?

Step 0: Get your Treasure Data API Key

Please retrieve your Write-Only API key from here.

Step 1: Prepare Configuration File

Please prepare your configuration file: specify your TD Write-Only API key, the destination database and table name. An example configuration is available here.

# Treasure Data Output
# ===================
# This configuration file specifies the information to be used
# when flushing information to TreasureData.com service. All
# key fields in the 'TD' section are mandatory.
[TD]
    # API
    # ===
    # The TreasureData API key. To obtain this please log into your
    # TreasureData Console:
    #
    # 1. Go to https://console.treasuredata.com/users/current
    # 2. API Keys box: copy the API key hash
    API YOUR_WRITE_ONLY_API_KEY

    # Database
    # ========
    # Specify the name of your database, it must exists.
    Database db_example

    # Table
    # =====
    # Specify the database table name where the records will be stored
    Table    table_example

Step 2: Embed libfluent-bit into your Applications

Please get the entire source code from GitHub.

When Fluent Bit is built, it creates the final binary and a shared library version called libfluent-bit.so which can be linked in your projects. The library allows you to enqueue data into the engine and return as quickly as possible. All the hard work is done in one posix thread that works in an asynchronous mode and your caller process or thread will not be blocked.

Here’s an example program to send the data to Treasure Data.

#include <unistd.h>
#include <fluent-bit.h>

int main(int argc, char **argv)
{
    int i;
    int n;
    int ret;
    char tmp[256];
    struct flb_config *config;

    if (argc < 2) {
        fprintf(stderr, "Usage: td /path/to/configuration.file\n");
        exit(EXIT_FAILURE);
    }

    /* Create configuration context */
    config = flb_config_init();
    if (!config) {
        exit(EXIT_FAILURE);
    }

    /* Enable verbose messages */
    flb_config_verbose(FLB_TRUE);

    /* Load a configuration file (required by TD output plugin) */
    ret = flb_lib_config_file(config, argv[1]);

    /* Initialize library */
    ret = flb_lib_init(config, (char *) "td");
    if (ret != 0) {
        exit(EXIT_FAILURE);
    }

    /* Start the background worker */
    flb_lib_start(config);

    /* Push some data */
    for (i = 0; i < 100; i++) {
        n = snprintf(tmp, sizeof(tmp) - 1, "{\"key\": \"val %i\"}", i);
        flb_lib_push(config, tmp, n);
    }

    flb_lib_stop(config);
    return 0;
}

As you can see, we only need to include the fluent-bit.h header and use the functions in the right order. The description of each function is available here:

Further Information

More information is available in the following links:


Last modified: Feb 24 2017 09:41:25 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.