json parser plugin for Data Connector

The json parser plugin for Data Connector parse json data. It pulgin takes one optional option as explained below:

Option Description
stop_on_invalid_record Stop bulk load transaction if a file includes invalid record (such as invalid json). (false by default)

Table of Contents

Configuration

It’s set automatically by guess command for generating load.yml. Or set the parser section’s type json in load.yml, for example:

in:
...
parser:
  type: json
out:
...

Use Case Example: Amazon S3 to Treasure Data

When your source file is json data like this sample.

Example of Source data

{"time":1455829282,"ip":"93.184.216.34","name":"frsyuki"}
{"time":1455829282,"ip":"172.36.8.109":"sadayuki"}
{"time":1455829284,"ip":"example.com","name":"Treasure Data"}
{"time":1455829282,"ip":"10.98.43.1","name":"MessagePack"}

load.yml example is this.

in:
type: s3
access_key_id: <access key>
secret_access_key: <secret access key>
bucket: <bucket name>
path_prefix: <file path>
parser:
  type: json
out: {mode: append}
exec: {}

You can show parsed json data which is named record column by preview command.

$ td connector:preview load.yml
+---------------------------------------------------------------------------+
| record:json                                                               |
+---------------------------------------------------------------------------+
| "{\"ip\":\"93.184.216.34\",\"time\":1455829282,\"name\":\"frsyuki\"}"     |
| "{\"ip\":\"172.36.8.109\",\"time\":1455829282,\"name\":\"sadayuki\"}"     |
| "{\"ip\":\"example.com\",\"time\":1455829284,\"name\":\"Treasure Data\"}" |
| "{\"ip\":\"10.98.43.1\",\"time\":1455829282,\"name\":\"MessagePack\"}"    |
+---------------------------------------------------------------------------+

Import to Treasure Data table.

$ td connector:issue load.yml --database <database name> --table <table name>  --auto-create-table

Result table is this.

The json data shows string type, but you can use JSON Functions as below.


Last modified: May 25 2016 01:45:40 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.