You can write job results from Treasure Data directly to a web server or HTTP API that accepts a PUT request.

Continue to the following topics:

Prerequisites

  • Basic knowledge of Treasure Data, including the TD Toolbelt.

  • A web server configured to accept query results as PUT requests.


Use the TD Console to Create Your Connection

You can use the TD Console to configure your connection.

Create a New Connection

Go to TD Catalog and search and select <integration>.

You can control access to your endpoint by using a list of static IPs. Contact support@treasuredata.com if you are interested.

Get Your Web Server Connection Details

Obtain your credential to access your HTTP put endpoint if any are required.

Write the Query

Go to the TD Console query editor page and compile your query.

Specify the Result Export Details.

After your query setup is complete, select Export Results located at the top right of your query editor.


The Choose Integration dialog opens. Type the connection name in the search box to filter and select your connection. In the Create Integration pane, select HTTP PUT from the drop-down menu. Edit all the information, including your Host, Port, and username and password if applicable.

If a port number is not specified, the default value is 80 for HTTP and 443 for HTTPS. If username and password are not given, then authentication is not used.


Select Save and Continue.

Set Transfer Settings

After setting up your connection, set the transfer parameters. Enter the endpoint path and parameter information.

Domain Field Configuration

Enter only the domain information in the host field. Do not include the URL path in the host field. The following examples show endpoint URLs and their respective values.

Example endpoint #1: http://www.yourdomain.com:80/api/custom_resources/resources

Host: www.yourdomain.com

Port: 80

Path & Params: /api/custom_resources/resources


Example endpoint #2: http://testapis.co/rest/api

Host:testapis.co

Path & Params: /rest/api


Example endpoint #3: http://www.randomapi.com/rest/api?parameter1

Host:www.randomapi.com

Path & Params: /rest/api?parameter1

Execute the Query

Either save the query with a name and run or just run the query. Upon successful completion of the query, the query result is automatically exported to the endpoint specified using a PUT request.

Usage from CLI

You can also use CLI for Result Export to an HTTP endpoint.

For On-Demand Jobs

For on-demand jobs, just add the --result option to the td query command. After the job is finished, the results are sent to the specified URL via a PUT request.

$ td query \
  --result 'web://domain.com/path' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Or doing POST request with method=post query parameter.

$ td query \
  --result 'web://domain.com/path?method=post' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Here is another example URL. Basic HTTP authentication and port specification are supported.

web://user:pass@domain.com:8080/path1/path2

We also support transfer via HTTPS if you use webs as the protocol header instead of web. For example:

webs://securedomain.com/path1/path2

If a port number is not specified, the default value is 80 for HTTP and 443 for HTTPS. If username and password are not given, then authentication is not used.

The request body is a JSON with the following fields: column_names, column_types, and data (the result of the query). An example of what the query above may output is shown below.

{ "column_names": [ "code", "cnt" ],
  "column_types": [ "string", "long" ],
  "data": [ [ "200", 4981 ], [ "500", 2 ], [ "404", 17 ] ] }

The maximum number of rows is 100,000. If the result exceeds 100,000 rows, the remaining rows are thrown away.

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. After every job run, the results are sent via PUT requests in the same manner as on-demand jobs.

$ td result:create myweb web://domain.com/
$ td sched:create hourly_count_example "0 * * * *" \
  -d testdb \
  --result myweb:path \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"
  • No labels