Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.

This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.

What can you do with this Integration?

• Create buckets: Create and name a bucket that stores data.
• Storing data: Store an infinite amount of data in a bucket.

Differences between Amazon S3 Export Integration v2 and Amazon S3 Export Integration v1

Review information in the following table to understand the differences and potential advantages between v2 and v1.

This topic includes:

Prerequisites

• Basic knowledge of Treasure Data, including the TD Toolbelt.

• For AWS: the IAM User

  • with s3:PutObject, s3:AbortMultipartUpload permissions 

  • with kms:Decrypt, kms:GenerateDataKey* permissions when selecting sse-kms setting

Requirements and Limitations

• The default query result limit for export to S3 is 100GB.  you could config part size setting up to 5000 (MB), the file limit will be about 5TB.

• The default export format is CSV RFC 4180.

• Output in TSV, JSONL format is also supported.

About S3 Server-Side Encryption

You can encrypt upload data with AWS S3 Server-Side Encryption. You don’t need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).

Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have server-side encryption enabled, you don't have to turn on the SSE option. However, job results might fail if you have bucket policies to reject HTTP requests without encryption information.

About KMS Server-Side Encryption

You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3). 

When you enable AWS KMS for server-side encryption in Amazon S3

• if not input KMS key id, it will create/using the default KMS key

• if input KMS Key  ID, you must choose symmetric CMK, not asymmetric CMKs

• The AWS KMS CMK must be in the same Region as the bucket

About File Formats for S3

code,cnt
200,4981
302,
404,17
500,2

code|cnt
200|4981
302|NULL
404|17
500|2

{"code": 200, "cnt": 4981}
{"code": 302, "cnt": null}
{"code": 404, "cnt": 17}
{"code": 500, "cnt": 2}

Use the TD Console to Create a Connection

In Treasure Data, you must create and configure the data connection prior to running your query. As part of the data connection, you provide authentication to access the integration.

Create a New Authentication

1. Open TD Console.

2. Navigate to Integrations Hub > Catalog.

3. Search for S3 and select AmazonS3.

4. Select Create Authentication.

5. Type the credentials to authenticate:

Parameter	Description
Endpoint	S3 service endpoint override. You can find region and endpoint information from AWS Document. (Ex. s3.ap-northeast-1.amazonaws.com) When specified will override region setting
Region	AWS Region
Authentication Method	basic	Uses access_key_id and secret_access_key to authenticate. See AWS Programmatic access. Access Key ID Secret access key
	session (Recommended)	Uses temporary-generated access_key_id, secret_access_key and session_token. Access Key ID Secret access key Secret token
	assume_role	Uses role access. See AWS AssumeRole TD's Instance Profile Account ID Your Role Name External ID Duration In Seconds
	anonymous	Not Support
Access Key ID	AWS S3 issued
Secret Access Key	AWS S3 issued

Create authentication with the assume_role authentication method 

1. Create a new authentication with the assume_role authentication method
   
2. Create your AWS IAM role
   
   
   

6. Select Continue. 

7. Type a name for your connection.

8. Select Done.

Define your Query

Integration Export Parameters for S3 

Example Query

SELECT * FROM www_access

(Optional) Schedule the Query

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

 *    *    *    *    *
 -    -    -    -    -
 |    |    |    |    |
 |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 |    |    |    +---------- month (1 - 12)
 |    |    +--------------- day of month (1 - 31)
 |    +-------------------- hour (0 - 23)
 +------------------------- min (0 - 59)

(Optional) Configure Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Exporting Data with Parameters.

S3 (v2) Configuration Keys

sse_algorithm

AES256

kms_key_id

csv, tsv, jsonl

Example Workflow for S3 (v2)

_export:
  td:
  database: td.database

+s3v2_test_export_task:
  td>: export_s3v2_test.sql
  database: ${td.database}
  result_connection: s3v2_conn
  result_settings:
  	bucket: my-bucket
  	path: /path/to/target.csv
  	sse_type: sse-s3
  	format: csv
  	compression: gz
  	header: false
    delimiter: default
    null_value:  empty
    newline: LF
  	quote_policy: MINIMAL
  	escape: '"'
  	quote: '"'
  	part_size: 20

(Optional) Configure Export Results in CLI

To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3
You can specify detailed settings to export your S3 via --result parameter. 

Only support create authentication with Assume Role using the console, creating through TD CLI will result in an error

Example for CLI command for S3 (v2)

td query \
--result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto