Page tree
Skip to end of metadata
Go to start of metadata

Treasure Data users can ingest data through the public REST API. You can use Treasure Data to create custom webhooks into your data. This article describes the specification of the data import REST API.

REST API to control databases, tables, jobs, and such, refer to REST APIs in Treasure Data.



Prerequisites

  • Basic knowledge of Treasure Data


Endpoint

All API requests should be sent to the appropriate endpoint for your region. 

The Postback GET endpoint will be deprecated in 2022. Due to changes in the backend, the GET endpoint may incur data loss. Use the Postback POST endpoint instead.


Limitation

We do not throttle access at the moment. However, we might limit access to our API, if deemed necessary to ensure the health of our system.

POST /postback/v3/event/{database}/{table}

Authentication

Every request must contain authentication information. Authentication can be provided in two ways:

  • specifying X-TD-Write-Key HTTP header. The header format is as follows:
    ‘X-TD-Write-Key: <APIKEY>’

  • specifying the td_write_key URL parameter. For example:
    /postback/v3/event/{database}/{table}?td_write_key=<APIKEY>
    The API key should be URL encoded as all other key/values parameters.

In case both header and URL parameters are provided, the header takes precedence.

The API key can be retrieved from the TD Console. It’s recommended to use Write Only API Keys.

Parameters

  • {database}: specify the destination database name in URL. only alphabets/numbers or _ are allowed.

  • {table}: specify destination table name in URL. only alphabets/numbers or _ are allowed.

Request Body

Specify JSON as a body. The JSON content immediately becomes a record in Treasure Data. Also include Content-Type: application/json in HTTP request headers.

Response

  • HTTP 200 : Success

  • HTTP 400 : Bad Request

  • HTTP 403 : Forbidden


Example

Here’s the example curl command to import the record via POST method.

$ curl -D a -X POST \
  -H 'Content-Type: application/json' \
  -H 'X-TD-Write-Key: XXXX' \
  --data-binary '{"param1":"value1", "param2":1234}' \
  https://in.treasuredata.com/postback/v3/event/test_db/test_tbl
{}
$ cat a
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 03 Dec 2015 02:31:32 GMT
Content-Length: 2
Connection: keep-alive

This will ingest the records as shown:

{
  'time': '1448344701',
  'param1': 'value1',
  'param2': 1234
}


GET /postback/v3/event/{database}/{table}

The Postback GET endpoint will be deprecated in 2022. Due to changes in the backend, the GET endpoint may incur data loss. Use the Postback POST endpoint instead.

Authentication

Every request must contain authentication information, provided by the td_write_key URL parameter. The api key can be retrieved from the TD Console. It’s recommended to use Write Only API Keys.

https://in.treasuredata.com/postback/v3/event/{database}/{table}?td_write_key=XXXYYYZZZZ&...

Parameters

Specify these two parameters with a path.

  • {database}: specify the destination database name in URL. only alphabets/numbers or _ are allowed.

  • {table}: specify destination table name in URL. only alphabets/numbers or _ are allowed.

Any URL parameters, except special parameters listed as follows, is considered as a column inside the record.

  • td_write_key: Already described above.

  • td_record_time: Specify UNIX timestamp value, to set time column in Treasure Data. If it’s not specified, the time records were received is used instead. Received timestamps that older than 7 days, and newer than 3 days ahead of the current date are ignored.

  • td_global_id: If we add td_global_id=td_global_id in the query string of postback url, API add a column named td_global_id with value of actual td_global_id into the record.

  • td_ip: If we add td_ip=td_ip in the query string of postback url, API add a column named td_ip with value of source IP into the record.

  • td_ua: If we add td_ua=td_ua in the query string of postback url, API add a column named td_ua with value of actual user-agent into the record.

Parameter values are processed as the data type string, and cannot be changed at the ingestion time. That’s why it is recommended that you use POST, where you can, to specify records as JSON with types.

Response

  • HTTP 200 : Success

  • HTTP 400 : Bad Request

  • HTTP 403 : Forbidden

Example

The following example shows the curl command used to import the record via GET method.

$ curl -D a \
  "https://in.treasuredata.com/postback/v3/event/test_db/test_tbl?td_write_key=XXXYYYZZZ&param1=value1&param2=value2"
{}
$ cat a
HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 09 Jan 2016 01:08:21 GMT
Content-Length: 2
Connection: keep-alive

Records are ingested, as shown:

{
  'time': '1448344701',
  'param1': 'value1',
  'param2': 'value2'
}

When you try the preceding example, change the time value to current unix time value. Records that have a timestamp older than 7 days, or are 3 days ahead of the current date are ignored.

  • No labels