Writing Job Results into your TD Table.
This article explains how to write job results to your Treasure Data (TD) table.
Table of Contents
- Basic knowledge of Treasure Data, including the toolbelt.
For On-demand Jobs
For on-demand jobs, just add the
--result option to the
td query command. Once the job is finished, the results are written into
output_table within the
$ td query --result 'td://@/testdb/output_table' \ -w -d testdb \ "SELECT code, COUNT(1) FROM www_access GROUP BY code"
For Scheduled Jobs
For scheduled jobs, just add the
--result option when scheduling a job. Every time the job runs, the results are written into
$ td result:create mydb td://@/testdb $ td sched:create hourly_count_example "0 * * * *" \ -d testdb "select count(*) from www_access" \ --result mydb:output_table
Two Ways to Modify Data (Append/Replace)
You can add or delete data in two ways.
td://@/testdb/output_table # append td://@/testdb/output_table?mode=append # append td://@/testdb/output_table?mode=replace # replace
This is the default mode. The query results are appended to the table. If the table does not exist yet, a new table will be created. This operation is not idempotent.
If the table already exists, the rows of the existing table are replaced with the query results. If the table does not exist yet, a new table will be created.
We achieve atomicity (so that a consumer of the table always has consistent data) by performing the following three steps in a single transaction.
- Create a temporary table.
- Write to the temporary table.
- Replace the existing table with the temporary table atomically.
Last modified: Feb 24 2017 09:27:52 UTC