Writing Job Results to a Web Server

This article explains how to send job results directly to your web server.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data, including the toolbelt.
  • A web server configured to accept query results as PUT requests.

Basic Usage



Untitled-3
Our Premium plan includes advanced security features, which includes a list of static IPs Treasure Data is using. You can limit the access to your endpoint by using these IPs. Please contact support@treasuredata.com if you need it too.

For On-demand Jobs

For on-demand jobs, just add the --result option to the td query command. After the job is finished, the results are sent to the specified URL via a PUT request.

$ td query \
  --result 'web://domain.com/path' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Or doing POST request with method=post query parameter.

$ td query \
  --result 'web://domain.com/path?method=post' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Here is another example URL. Basic HTTP authentication and port specification are supported.

web://user:pass@domain.com:8080/path1/path2

We also support transfer via HTTPS if you use webs as the protocol header instead of web. For example:

webs://securedomain.com/path1/path2
Untitled-3
If a port number is not specified, the default value is 80 for HTTP and 443 for HTTPS. If username and password are not given, then authentication is not used.

The request body is a JSON with the following fields: column_names, column_types, and data (the result of the query). An example of what the query above may output is shown below.

{ "column_names": [ "code", "cnt" ],
  "column_types": [ "string", "long" ],
  "data": [ [ "200", 4981 ], [ "500", 2 ], [ "404", 17 ] ] }
Untitled-3
The maximum number of rows is 100,000. If the result exceeds 100,000 rows, the remaining rows are thrown away.

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. After every job run, the results are sent via PUT requests in the same manner as on-demand jobs.

$ td result:create myweb web://domain.com/
$ td sched:create hourly_count_example "0 * * * *" \
  -d testdb \
  --result myweb:path \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Last modified: Jul 11 2016 05:57:04 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.