Excerpt | ||
---|---|---|
| ||
Keep the introduction brief. The customer is looking for this integration because they already have the product. Example: Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements. |
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.
Excerpt | ||
---|---|---|
| ||
Replace the product name in the following sentence: Example: This TD export integration allows you to write job results from Treasure Data directly to <product name>. |
This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.
What can you do with this Integration?
Excerpt | ||
---|---|---|
| ||
The Product Manager needs to add two to three bullets to explain how the customer can benefit from this integration. Example:
|
- Create buckets: Create and name a bucket that stores data.
- Storing data: Store an infinite amount of data in a bucket.
Differences between Amazon S3 Export Integration v2 and Amazon S3 Export Integration v1
Review information in the following table to understand the differences and potential advantages between v2 and v1.
Feature | Amazon S3 v2 | Amazon S3 v1 |
---|---|---|
Server-side Encryption with keys | X | |
Server-side Encryption with Customer Master Key (CMK) stored in AWS Key Management Service | X | |
Support for output in JSONL format | X | |
Support for output and query results output in JSONL format | X | |
Output data size limit = 100GB | X | |
Output data size limit = 5TB | X | |
Removal of Anonymous mode from the authentication method | X | |
No support for Quote Policy for output data format | X | |
Support for Quote Policy for output data format | X |
This topic includes:
Table of Contents maxLevel 3 exclude What can you do with this Integration?
Prerequisites
Excerpt | ||
---|---|---|
| ||
Include a bulleted list of prerequisites the customer needs to successfully implement this export integration. Example:
|
Basic knowledge of Treasure Data, including the TD Toolbelt.
For AWS: the IAM User
with s3:PutObject, s3:AbortMultipartUpload permissions
with kms:Decrypt, kms:GenerateDataKey* permissions when selecting sse-kms setting
Requirements and Limitations
Excerpt | ||
---|---|---|
| ||
List any product or integration limitations that affect the function of the integration. Example:
|
The default query result limit for export to S3 is 100GB. you could config part size setting up to 5000 (MB), the file limit will be about 5TB.
The default export format is CSV RFC 4180.
Output in TSV, JSONL format is also supported.
About S3 Server-Side Encryption
You can encrypt upload data with AWS S3 Server-Side Encryption. You don’t need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).
Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have server-side encryption enabled, you don't have to turn on the SSE option. However, job results might fail if you have bucket policies to reject HTTP requests without encryption information.
About KMS Server-Side Encryption
You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3).
When you enable AWS KMS for server-side encryption in Amazon S3
if not input KMS key id, it will create/using the default KMS key
if input KMS Key ID, you must choose asymmetric CMK, not asymmetric CMKs
The AWS KMS CMK must be in the same Region as the bucket
About File Formats for S3
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
For both CSV, TSV, JSONL formats, the following table lists options you can use to customize the final format of the files written into the destination:
|
Use the TD Console to Create a Connection
In Treasure Data, you must create and configure the data connection prior to running your query. As part of the data connection, you provide authentication to access the integration.
Create a New Authentication
Numbered Headings | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||
Open TD Console.Navigate to Integrations Hub > Catalog.Search for S3 and selectSelect Create Authentication.Type the credentials to authenticate:
Select Continue.Type a name for your connection.Select Done. |
Define your Query
Numbered Headings |
---|
|
IntegratioExportn Parameters for S3
Parameter | Data Type | Required? | Supported in V1? | Description |
---|---|---|---|---|
Server-side Encryption | String | yes, only sse-s3 | Support values:
| |
Server-side Encryption Algorithm | String | yes | Support value:
| |
KMS Key ID | String | no | Symmetric AWS KMS Key Id, if not input KMS key id, it will create/using the default KMS key | |
Bucket | String | yes | yes | Provide the S3 bucket name (Ex. your_bucket_name) |
Path | String | yes | yes | Specify s3 filename (object key), include an extension (Ex. test.csv) |
Format | String | yes | Format of the exported file: csv, tsv, jsonl | |
Compression | String | yes | The compression format of the exported files (Ex. None or gz) | |
Header | Boolean | yes | Include header in the exported file | |
Delimiter | String | yes | Use to specify the delimiter character (Ex, (comma)) | |
String for NULL values | String | yes | Placed holder to insert for null values (Ex. Empty String) | |
End-of-line character | String | yes | Specify the EOL(End-Of-Line) representation (Ex. CRLF, LF) | |
Quote Policy | String | no | Use to determine field type to quote. Support values:
Default value: MINIMAL | |
Quote character (Optional) | Char | yes | The character used for quotes in the exported file(Ex. "). Only quote those fields which contain delimiter, quote, or any of the characters in lineterminator. If the input is more than 1 character, the default value will be used | |
Escape character(Optional) | Char | yes | The escape character is used in the exported file. If the input is more than 1 character, the default value will be used | |
Part Size (MB) (Optional) | Integer | no | The part size in multipart upload Default 10, min 5, max 5000 |
Example Query
Code Block | ||||
---|---|---|---|---|
| ||||
SELECT * FROM www_access |
(Optional) Schedule the Query
You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.
Numbered Headings | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
|
(Optional) Configure Export Results in Workflow
Within Treasure Workflow, you can specify the use of this data connector to export data.
Learn more at Using Workflows to Export Data with the TD Toolbelt.
S3 (v2) Configuration Keys
Name | Type | Required | Description |
---|---|---|---|
bucket | String | Yes | |
path | String | Yes | |
sse_type | String | sse-s3, sse-kms | |
sse_algorithm | String | AES256 | |
kms_key_id | String | ||
format | String | csv, tsv, jsonl | |
compression | String | none, gz | |
header | Boolean | Default true | |
delimiter | String | default , \t | | |
null_value | String | default, empty, \\N, NULL, null | |
newline | String | CR, LF, CRLF | |
quote_policy | String | ALL, MINIMAL, NONE | |
escape | Char | ||
quote | Char | ||
part_size | Integer |
Example Workflow for S3 (v2)
Code Block |
---|
_export: td: database: td.database +s3v2_test_export_task: td>: export_s3v2_test.sql database: ${td.database} result_connection: s3v2_conn result_settings: bucket: my-bucket path: /path/to/target.csv sse_type: sse-s3 format: csv compression: gz header: false delimiter: default null_value: empty newline: LF quote_policy: MINIMAL escape: '"' quote: '"' part_size: 20 |
(Optional) Configure Export Results in CLI
To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3
You can specify detailed settings to export your S3 via --result parameter.
Example for CLI command for S3 (v2)
Code Block | ||||
---|---|---|---|---|
| ||||
td query \ --result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \ -w -d testdb \ "SELECT 1 as col" -T presto |