C/C++ Data Import with Fluent Bit

You can collect data from server-side applications, games, and IoT devices that are based in C++. This article describes how to reliably import data from your C/C++ applications, using Fluent Bit.

Table of Contents

Prerequisites

  • Basic knowledge of C/C++ Application Development.
  • Basic knowledge of Treasure Data.

What is Fluent Bit?

Fluent Bit is a specialized open-source data collector tool for embedded operating systems and is developed by Treasure Data. Fluent Bit can be used as a standalone tool and can also be embedded in your C or C++ applications.

By emdedding Fluent Bit in your C/C++ applications, you can have robust and light-weight data collection mechanism (buffering, retry, etc) inside your applications.

How to Embed Fluent Bit and connect to Treasure Data?

Step 0: Get your Treasure Data API Key

Retrieve your Write-Only API key from here.

Step 1: Prepare Configuration File

Prepare your configuration file by specifying: your TD Write-Only API key, the destination database, and table name. An example configuration is available here.

# Treasure Data Output
# ===================
# This configuration file specifies the information to be used
# when flushing information to TreasureData.com service. All
# key fields in the 'TD' section are mandatory.
[TD]
    # API
    # ===
    # The TreasureData API key. To obtain the TD API Key log into your
    # Treasure Data Console:
    #
    # 1. Go to https://console.treasuredata.com/app/users/current
    # 2. API Keys box: copy the API key hash
    API YOUR_WRITE_ONLY_API_KEY

    # Database
    # ========
    # Specify the name of your database. The database must exist.
    Database db_example

    # Table
    # =====
    # Specify the database table name where the records will be stored
    Table    table_example

Step 2: Embed libfluent-bit into your Applications

Retrieve the entire source code from GitHub.

When Fluent Bit is built, it creates the final binary and a shared library version called libfluent-bit.so which can be linked in your projects. The library allows you to enqueue data into the engine and return quickly. All the hard work is done in one posix thread that works in an asynchronous mode. Your caller process or thread will not be blocked.

Here’s an example program to send the data to Treasure Data.

#include <unistd.h>
#include <fluent-bit.h>

int main(int argc, char **argv)
{
    int i;
    int n;
    int ret;
    char tmp[256];
    struct flb_config *config;

    if (argc < 2) {
        fprintf(stderr, "Usage: td /path/to/configuration.file\n");
        exit(EXIT_FAILURE);
    }

    /* Create configuration context */
    config = flb_config_init();
    if (!config) {
        exit(EXIT_FAILURE);
    }

    /* Enable verbose messages */
    flb_config_verbose(FLB_TRUE);

    /* Load a configuration file (required by TD output plugin) */
    ret = flb_lib_config_file(config, argv[1]);

    /* Initialize library */
    ret = flb_lib_init(config, (char *) "td");
    if (ret != 0) {
        exit(EXIT_FAILURE);
    }

    /* Start the background worker */
    flb_lib_start(config);

    /* Push some data */
    for (i = 0; i < 100; i++) {
        n = snprintf(tmp, sizeof(tmp) - 1, "{\"key\": \"val %i\"}", i);
        flb_lib_push(config, tmp, n);
    }

    flb_lib_stop(config);
    return 0;
}

You only need to include the fluent-bit.h header and use the functions in the right order. The description of each function is available here:

Further Information

More information is available in the following links:


Last modified: Oct 05 2017 17:05:13 UTC

If this article is incorrect or outdated, or omits critical information, let us know. For all other issues, access our support channels.