Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.

This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.

What can you do with this Integration?

• Create buckets: Create and name a bucket that stores data.
• Storing data: Store an infinite amount of data in a bucket.

Differences between Amazon S3 Export Integration v2 and Amazon S3 Export Integration v1

Review the information in the following table to understand the differences and potential advantages between v2 and v1.

This topic includes:

Prerequisites

• Basic knowledge of Treasure Data, including the TD Toolbelt.

• For AWS: the IAM User :

  • with s3:PutObject, s3:AbortMultipartUpload permissions.

  • with kms:Decrypt, kms:GenerateDataKey* permissions when selecting the sse-kms setting.

Requirements and Limitations

• The default query result limit for export to S3 is 100GB. You could config part size setting up to 5000 (MB), the file limit will be about 5TB.

• The default export format is CSV RFC 4180.

• Output in TSV, JSONL format is also supported.

Static IP Address of Treasure Data

The static IP address of Treasure Data is the access point and source of the linkage for this Integration. To determine the static IP address, contact your Customer Success representative or Technical support.

About S3 Server-Side Encryption

You can encrypt upload data with AWS S3 Server-Side Encryption. You don’t need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).

Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have the server-side encryption enabled, you don't have to turn on the SSE option. However, job results may fail if you have bucket policies to reject HTTP requests without encryption information.

About KMS Server-Side Encryption

You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3). 

When you enable AWS KMS for server-side encryption in Amazon S3:

• If you don't input the KMS Key ID, it will create/use the default KMS key.

• If you input the KMS Key ID, you must choose symmetric CMK (not asymmetric CMKs).

• The AWS KMS CMK must be in the same Region as the bucket.

About File Formats for S3

code,cnt
200,4981
302,
404,17
500,2

code|cnt
200|4981
302|NULL
404|17
500|2

{"code": 200, "cnt": 4981}
{"code": 302, "cnt": null}
{"code": 404, "cnt": 17}
{"code": 500, "cnt": 2}

Use the TD Console to Create a Connection

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

Create a New Authentication

Numbered Headings
start-numbering-with 1 start-numbering-at h5
Open TD Console. Navigate to Integrations Hub > Catalog. Search for S3 and select AmazonS3. Select Create Authentication. Type the credentials to authenticate: Parameter Description Endpoint S3 service endpoint override. You can find region and endpoint information from AWS Document. (Ex. s3.ap-northeast-1.amazonaws.com) When specified, it will override the region setting. Region AWS Region Authentication Method basic Uses access_key_id and secret_access_key to authenticate. See AWS Programmatic access. Access Key ID Secret access key session (Recommended) Uses temporary-generated access_key_id, secret_access_key and session_token. Access Key ID Secret access key Secret token assume_role Uses role access. See AWS AssumeRole TD's Instance Profile Account ID Your Role Name External ID Duration In Seconds anonymous Not Support Access Key ID AWS S3 issued Secret Access Key AWS S3 issued Info Create authentication with the assume_role authentication method  Create a new authentication with the assume_role authentication method Create your AWS IAM role Select Continue.  Type a name for your connection. Select Done.

Define your Query

Specify the Result Export Target

Create an Activation Using an Integration

Integration Export Parameters for S3 

1. Define any additional Export Results details. In your export integration content review the integration parameters.
   For example, your Export Results screen might be different, or you might not have additional details to fill out:
2. Select Done.
3. Run your query
4. Validate that your data moved to the destination you specified.

Example Query

SELECT * FROM www_access

(Optional) Schedule the Query

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

(Optional) Configure Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Exporting Data with Parameters.

S3 (v2) Configuration Keys

sse_algorithm

AES256

kms_key_id

csv, tsv, jsonl

Example Workflow for S3 (v2)

_export:
  td:
  database: td.database

+s3v2_test_export_task:
  td>: export_s3v2_test.sql
  database: ${td.database}
  result_connection: s3v2_conn
  result_settings:
  	bucket: my-bucket
  	path: /path/to/target.csv
  	sse_type: sse-s3
  	format: csv
  	compression: gz
  	header: false
    delimiter: default
    null_value:  empty
    newline: LF
  	quote_policy: MINIMAL
  	escape: '"'
  	quote: '"'
  	part_size: 20

(Optional) Configure Export Results in CLI

To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3.
You can specify detailed settings to export your S3 via the --result parameter. 

Creating authentication with Assume Role is only supported through the console. Attempting to create it through the TD CLI will result in an error.

Example CLI Command for S3 (v2)

td query \
--result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto