Writing Job Results into your TD Table.

This article explains how to write job results to your Treasure Data (TD) table.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data, including the toolbelt.

Basic Usage



For On-demand Jobs

For on-demand jobs, just add the --result option to the td query command. Once the job is finished, the results are written into output_table within the testdb database.

$ td query --result 'td://@/testdb/output_table' \
  -w -d testdb \
  "SELECT code, COUNT(1) FROM www_access GROUP BY code"      

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. Every time the job runs, the results are written into output_table.

$ td result:create mydb td://@/testdb
$ td sched:create hourly_count_example "0 * * * *" \
  -d testdb "select count(*) from www_access" \
  --result mydb:output_table

Two Ways to Modify Data (Append/Replace)

You can add or delete data in two ways.

td://@/testdb/output_table              # append
td://@/testdb/output_table?mode=append  # append
td://@/testdb/output_table?mode=replace # replace

APPEND

This is the default mode. The query results are appended to the table. If the table does not exist yet, a new table will be created. This operation is not idempotent.

REPLACE

If the table already exists, the rows of the existing table are replaced with the query results. If the table does not exist yet, a new table will be created.

We achieve atomicity (so that a consumer of the table always has consistent data) by performing the following three steps in a single transaction.

  1. Create a temporary table.
  2. Write to the temporary table.
  3. Replace the existing table with the temporary table atomically.

Last modified: Feb 24 2017 09:27:52 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.