...
This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.
This topic includes:
Table of Contents maxLevel 1
What can you do with this Integration?
...
| Feature | Amazon S3 v2 | Amazon S3 v1 |
|---|---|---|
| Server-side Encryption with Customer Master Key (CMK) stored in AWS Key Management Service | X | |
| Support for Quote Policy for output data format | X | |
| Support Assume Role authentication method | X |
This topic includes:
Table of Contents maxLevel 1
Prerequisites
Basic knowledge of Treasure Data, including the TD Toolbelt.
For AWS: the IAM User :
with s3:PutObject, s3:AbortMultipartUpload permissions.
with kms:Decrypt, kms:GenerateDataKey* permissions when selecting the sse-kms setting.
...
| Numbered Headings | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||
Select Export Results. |
| Field | Description |
|---|---|
| Is user directory Root? | If selected, the user directory is treated as the root directory. (ex. ‘/home/treasure-data’ as ‘/’) |
| Path prefix: | The file path where the file will be stored. |
| Rename file after upload finish | If selected, SFTP result output renames the file on the remote SFTP server from “.xxx.tmp” to “.xxx” after all the data is transferred. Some MA tools try to import data when a file with a specific name exists on the SFTP server. The temp name option is useful for such cases. |
| Format | The format of the exported files:
|
| Compression | The compression format of the exported files:
|
| Header line? | The header line with column name as the first line. |
| Delimiter | The delimited character:
|
| Quote policy | The policy for a quote:
|
| Null string | How null value of the result of the query displays:
|
| End-of-line character | The EOL (end-of-line) character:
|
| Temp filesize threshold | The maximum file size (in bytes) of a local temp file. When the temp file reaches the threshold, the file flushes to a remote file. If you encounter the error `channel is broken`, reduce the value of this option to resolve the error. |
...
- Define any additional Export Results details . In your export integration and content review the integration parameters.
For example, your Export Results screen might be different, or you might not have additional details to fill out: .
- Select Done.
- Run your query
- Validate that your data moved to the destination you specified.
...
| Parameter | Data Type | Required? | Supported in V1? | Description |
|---|---|---|---|---|
| Server-side Encryption | String | yes, only sse-s3 | Support values:
| |
| Server-side Encryption Algorithm | String | yes | Support value:
| |
| KMS Key ID | String | no | Symmetric AWS KMS Key ID. If there is no input for the KMS Key ID, it will create/use the default KMS Key. | |
| Bucket | String | yes | yes | Provide the S3 bucket name (Ex., your_bucket_name). |
| Path | String | yes | yes | Specify the s3 filename (object key), and include an extension (Ex. test.csv). |
| Format | String | yes | Format of the exported file: csv, tsv, jsonl | |
| Compression | String | yes | The compression format of the exported files (Ex., None or gz) | |
| Header | Boolean | yes | Include a header in the exported file. | |
| Delimiter | String | yes | Use to specify the delimiter character (Ex., (comma)) | |
| String for NULL values | String | yes | Placed holder to insert for null values (Ex. Empty String) | |
| End-of-line character | String | yes | Specify the EOL(End-Of-Line) representation (Ex. CRLF, LF) | |
| Quote Policy | String | no | Use to determine field type to quote. Support values:
Default value: MINIMAL | |
| Quote character (Optional) | Char | yes | The character used for quotes in the exported file (Ex. "). Only quote those fields which contain the delimiter, quote, or any of the characters in the line terminatorlineterminator. If the input is more than 1 character, the default value will be used. | |
| Escape character(Optional) | Char | yes | The escape character is used in the exported file. If the input is more than 1 character, the default value will be used. | |
| Part Size (MB) (Optional) | Integer | no | The part size in multipart upload. Default 10, min 5, max 5000 |
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
SELECT * FROM www_access |
(Optional) Schedule
...
Query Export Jobs
You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.
...
| Code Block |
|---|
_export:
td:
database: td.database
+s3v2_test_export_task:
td>: export_s3v2_test.sql
database: ${td.database}
result_connection: s3v2_conn
result_settings:
bucket: my-bucket
path: /path/to/target.csv
sse_type: sse-s3
format: csv
compression: gz
header: false
delimiter: default
null_value: empty
newline: LF
quote_policy: MINIMAL
escape: '"'
quote: '"'
part_size: 20 |
(Optional)
...
Export
...
Integration Using the CLI
To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3.
You can specify detailed settings to export your S3 via the --result parameter.
...