Skip to content
/
Web Server and HTTP Put E...
Last updated

Web Server and HTTP Put Endpoint Export Integration

You can write job results from Treasure Data directly to a web server or HTTP API that accepts a PUT request.

Prerequisites

  • Basic knowledge of Treasure Data, including the TD Toolbelt.
  • A web server configured to accept query results as PUT requests.

Static IP Address of Treasure Data Integration

If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.

Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/

Use the TD Console to Create Your Connection

You can use the TD Console to configure your connection.

Create a New Connection

Go to TD Catalog; search and select the integration.

Get Your Web Server Connection Details

Obtain your credentials to access your HTTP endpoint if any are required.

Write the Query

Go to the TD Console query editor page and compile your query.

Specify the Result Export Details.

After your query setup is complete, select Export Results located at the top right of your query editor.

The Choose Integration dialog opens. Type the connection name in the search box to filter and select your connection. In the Create Integration pane, select HTTP PUT from the drop-down menu. Edit all the information, including your Host, Port, and username and password if applicable.

If a port number is not specified, the default value is 80 for HTTP and 443 for HTTPS. If username and password are not given, then authentication is not used.

Select Save and Continue.

Set Transfer Settings

After setting up your connection, set the transfer parameters. Enter the endpoint path and parameter information.

Domain Field Configuration

Enter only the domain information in the host field. Do not include the URL path in the host field. The following examples show endpoint URLs and their respective values.

Example endpoint #1: http://www.yourdomain.com:80/api/custom_resources/resources

Host: www.yourdomain.com

Port: 80

Path & Params: /api/custom_resources/resources

Example endpoint #2: http://testapis.co/rest/api

Host:testapis.co

Path & Params: /rest/api

Example endpoint #3: http://www.randomapi.com/rest/api?parameter1

Host:www.randomapi.com

Path & Params: /rest/api?parameter1

Execute the Query

Either save the query with a name and run or just run the query. Upon successful completion of the query, the query result is automatically exported to the endpoint specified using a PUT request.

Activate a Segment in Audience Studio

You can also send segment data to the target platform by creating an activation in the Audience Studio.

  1. Navigate to Audience Studio.
  2. Select a parent segment.
  3. Open the target segment, right-mouse click, and then select Create Activation.
  4. In the Details panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters.
  5. Customize the activation output in the Output Mapping panel.

  • Attribute Columns
    • Select Export All Columns to export all columns without making any changes.
    • Select + Add Columns to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select + Add Columnsto add new columns for your activation output.
  • String Builder
    • + Add string to create strings for export. Select from the following values:
      • String: Choose any value; use text to create a custom value.
      • Timestamp: The date and time of the export.
      • Segment Id: The segment ID number.
      • Segment Name: The segment name.
      • Audience Id: The parent segment number.
  1. Set a Schedule.

  • Select the values to define your schedule and optionally include email notifications.
  1. Select Create.

If you need to create an activation for a batch journey, review Creating a Batch Journey Activation.

Usage from CLI

You can also use CLI for Result Export to an HTTP endpoint.

For On-Demand Jobs

For on-demand jobs, just add the --result option to the td query command. After the job is finished, the results are sent to the specified URL via a PUT request.

$ td query 
  --result 'web://domain.com/path' 
  -w -d testdb 
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Or doing POST request with method=post query parameter.

$ td query 
  --result 'web://domain.com/path?method=post' 
  -w -d testdb 
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

Here is another example URL. Basic HTTP authentication and port specification are supported.

web://user:pass@domain.com:8080/path1/path2

We also support transfer via HTTPS if you use webs as the protocol header instead of web. For example:

webs://securedomain.com/path1/path2

If a port number is not specified, the default value is 80 for HTTP and 443 for HTTPS. If username and password are not given, then authentication is not used.

The request body is a JSON with the following fields: column_names, column_types, and data (the result of the query). An example of what the query above may output is shown below.

{ "column_names": [ "code", "cnt" ],
  "column_types": [ "string", "long" ],
  "data": [ [ "200", 4981 ], [ "500", 2 ], [ "404", 17 ] ] }

The maximum number of rows is 100,000. If the result exceeds 100,000 rows, the remaining rows are thrown away.

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. After every job run, the results are sent via PUT requests in the same manner as on-demand jobs.

$ td result:create myweb web://domain.com/
$ td sched:create hourly_count_example "0 * * * *" 
  -d testdb 
  --result myweb:path 
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"
Previous page
Next page