msgpack parser plugin for Data Connector

The msgpack parser plugin for Data Connector parse msgpack data. It accepts 3 options as explained below:

Option Description
row_encoding type of a row. “array” or “map” (enum, default: map)
file_encoding if a file includes a big array, set “array”. Otherwise, if a file includes sequence of rows, set “sequence” (enum, default: sequence)
columns Columns (schema, required)

The columns option declares the list of columns. This msgpack parser plugin ignores the header line.

Option Description
index Index number of the column (optional)
name Name of the column
type Type of the column (see below)

boolean: true or false
long: 64-bit signed integers
double: 64-bit floating point numbers
string: Strings
timestamp: Date and time with nano-seconds precision
json: JSON data
format Format of the timestamp if type is timestamp

Table of Contents

Configuration

You can set the parser section like an example:

in:
...
  parser:
    type: msgpack
    row_encoding: map
    file_encoding: sequence
    columns:
      - {index: 0, name: id, type: long}
      - {index: 1, name: name, type: string}
      - {index: 2, name: price, type: double}
      - {index: 3, name: tag, type: string}
      - {index: 4, name: timestamp, type: timestamp, format: '%Y-%m-%d'}
out:
...

Use Case Example: msgpack generated from csv

Example of Source data (before converting)

id,name,price,tag,timestamp
1,"A green door",11.50,"green","2016-01-02"
2,"A blue door",12.50,"blue","2016-01-03"
3,"A red door",13.50,"red","2016-01-04"
4,"A pink door",14.50,"pink","2016-01-05"
5,"A white door",15.50,"white","2016-01-06"
6,"A black door",16.50,"black","2016-01-07"
7,"A yellow door",17.50,"yellow","2016-01-08"
8,"A purple door",18.50,"purple","2016-01-09"

Example of load.yml

in:
...
parser:
   type: msgpack
   row_encoding: map
   file_encoding: sequence
   columns:
     - {index: 0, name: id, type: long}
     - {index: 1, name: name, type: string}
     - {index: 2, name: price, type: double}
     - {index: 3, name: tag, type: string}
     - {index: 4, name: timestamp, type: timestamp, format: '%Y-%m-%d'}
filters: []
out: {mode: append}
exec: {}

You can show parsed msgpack data by preview command.

$ td connector:preview load.yml
+---------+-----------------+--------------+------------+---------------------------+
| id:long | name:string     | price:double | tag:string | timestamp:timestamp       |
+---------+-----------------+--------------+------------+---------------------------+
| 1       | "A green door"  | 11.5         | "green"    | "2016-01-02 00:00:00 UTC" |
| 2       | "A blue door"   | 12.5         | "blue"     | "2016-01-03 00:00:00 UTC" |
| 3       | "A red door"    | 13.5         | "red"      | "2016-01-04 00:00:00 UTC" |
| 4       | "A pink door"   | 14.5         | "pink"     | "2016-01-05 00:00:00 UTC" |
| 5       | "A white door"  | 15.5         | "white"    | "2016-01-06 00:00:00 UTC" |
| 6       | "A black door"  | 16.5         | "black"    | "2016-01-07 00:00:00 UTC" |
| 7       | "A yellow door" | 17.5         | "yellow"   | "2016-01-08 00:00:00 UTC" |
| 8       | "A purple door" | 18.5         | "purple"   | "2016-01-09 00:00:00 UTC" |
+---------+-----------------+--------------+------------+---------------------------+

If your data doesn’t have a time column you may add it using add_time filter option. More details at add_time filter plugin

$ td connector:issue load.yml --database <database name> --table <table name>  --time-column timestamp --auto-create-table

Last modified: Feb 22 2017 23:40:42 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.