You can stream very large amounts of data from multiple devices into Treasure Data using a single MQTT broker. The MQTT broker works with your device management credentials to securely accept streaming data, and make the data available for queries in Treasure Data.

You can ingest data directly from individual devices into Treasure Data. Contact our Support team for more information.


Certification from your Device Manager and a Pipeline with MQTT Broker

You can use device management to authenticate your end devices and then push data into Treasure Data using device management. You configure your devices to send data to both your device manager and TD MQTT broker.

You can create multiple keys for your data collecting device but need only one authentication certificate.


For example, Pelion Device Management services are presented as one connection to the MQTT broker. The MQTT broker enables you to pipe the data collected from your Mbed Cloud to any destination that you want, including the Treasure Data platform.

You can stream device activities into TD. You create a query that captures device activity.

The following image shows a device panel on the right and the query in Treasure Data that captures device activity:


Prerequisites

  • Treasure Data Account (contact us if you have an interest in IoT Platform)

  • Devices from which you gather data

  • (Optional) Device Management system such as Pelion Device Management

  • Access to the TD MQTT broker endpoint

Get your Treasure Data API Key

Retrieve your Write-Only API key from your profile in Treasure Data.

Obtain your TD MQTT Broker endpoint

Treasure Data assigns an endpoint to your account. Read more about Treasure Data Endpoints.

Prepare your MQTT Client to Talk to MQTT Broker

TBD

Prepare Configuration File

Prepare your configuration file by specifying: your TD Write-Only API key, the destination database, and table name. An example configuration is as follows.

# Treasure Data Output
# ===================
# This configuration file specifies the information to be used
# when flushing information to TreasureData.com service. All
# key fields in the 'TD' section are mandatory.
[TD]
    # API
    # ===
    # The TreasureData API key. To obtain the TD API Key log into your
    # Treasure Data Console:
    #
    # 1. Go to https://console.treasuredata.com/users/current
    # 2. API Keys box: copy the API key hash
    API YOUR_WRITE_ONLY_API_KEY

    # Database
    # ========
    # Specify the name of your database. The database must exist.
    Database db_example

    # Table
    # =====
    # Specify the database table name where the records will be stored
    Table    table_example

Embed MQTT into your Device Management System

You can retrieve the source code example from GitHub.

Here’s an example code that you use to enable credentialing at your device management endpoint.

#include <unistd.h>
#include <fluent-bit.h>

int main(int argc, char **argv)
{
    int i;
    int n;
    int ret;
    char tmp[256];
    struct flb_config *config;

    if (argc < 2) {
        fprintf(stderr, "Usage: td /path/to/configuration.file\n");
        exit(EXIT_FAILURE);
    }

    /* Create configuration context */
    config = flb_config_init();
    if (!config) {
        exit(EXIT_FAILURE);
    }

    /* Enable verbose messages */
    flb_config_verbose(FLB_TRUE);

    /* Load a configuration file (required by TD output plugin) */
    ret = flb_lib_config_file(config, argv[1]);

    /* Initialize library */
    ret = flb_lib_init(config, (char *) "td");
    if (ret != 0) {
        exit(EXIT_FAILURE);
    }

    /* Start the background worker */
    flb_lib_start(config);

    /* Push some data */
    for (i = 0; i < 100; i++) {
        n = snprintf(tmp, sizeof(tmp) - 1, "{\"key\": \"val %i\"}", i);
        flb_lib_push(config, tmp, n);
    }

    flb_lib_stop(config);
    return 0;
}

Create Your Query

For example:



  • No labels