Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.

This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.


This topic includes:

What can you do with this Integration?

Differences between Amazon S3 Export Integration v2 and Amazon S3 Export Integration v1

Review the information in the following table to understand the differences and potential advantages between v2 and v1.

FeatureAmazon S3 v2Amazon S3 v1
Server-side Encryption with Customer Master Key (CMK)
stored in AWS Key Management Service
X
Support for Quote Policy for output data formatX
Support Assume Role authentication methodX

Prerequisites

Requirements and Limitations

Static IP Address of Treasure Data

The static IP address of Treasure Data is the access point and source of the linkage for this Integration. To determine the static IP address, contact your Customer Success representative or Technical support.

About S3 Server-Side Encryption

You can encrypt upload data with AWS S3 Server-Side Encryption. You don’t need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).

Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have the server-side encryption enabled, you don't have to turn on the SSE option. However, job results may fail if you have bucket policies to reject HTTP requests without encryption information.

About KMS Server-Side Encryption

You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3)

When you enable AWS KMS for server-side encryption in Amazon S3:

About File Formats for S3

For the CSV, TSV and JSONL formats, the following table lists the options you can use to customize the final format of the files written into the destination:

Name

Description

Restrictions

CSV default

TSV default

JSONL

Format

The main setting to specify the file format.


csv

csv (Use ‘tsv’ to select the TSV format)

Use JSONL to select JSONL format

Delimiter

Use to specify the delimiter character.


, (comma)

\t (tab)

Parameter ignored
Quote policyUse to determine field type to quote.
MINIMALMINIMALParameter ignored

Quote

Use to specify the quote character

Not available for TSV format

“ (double quote)

(no character)

Parameter ignored

Escape

Specifies the character used to escape other special characters.

Not available for TSV format

“ (double quote)

(no character)

Parameter ignored

Null

Use to specify how a ‘null’ value is displayed.


(empty string)

\N (backslash capital n)

Parameter ignored

Newline

Use to specify the EOL (End-Of-Line) representation.


\r\n (CRLF)

\r\n (CRLF)

\r\n (CRLF)

Header

Can be used to suppress the column header.


The column header is printed. Use ‘false’ to suppress.

The column header is printed. Use ‘false’ to suppress.

Parameter ignored


The following example shows a default sample output in CSV format when no customization is requested:

code,cnt
200,4981
302,
404,17
500,2


When the format=tsv, delimiter=|, and null=NULL options are specified. The output changes to:

code|cnt
200|4981
302|NULL
404|17
500|2


When the format=jsonl. The output changes to:

{"code": 200, "cnt": 4981}
{"code": 302, "cnt": null}
{"code": 404, "cnt": 17}
{"code": 500, "cnt": 2}



Use the TD Console to Create a Connection

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

Create a New Authentication

Open TD Console.
Navigate to Integrations Hub Catalog.
Search for S3 and select AmazonS3.
Select Create Authentication.
Type the credentials to authenticate:
ParameterDescription

Endpoint

S3 service endpoint override. You can find region and endpoint information from AWS Document. (Ex. s3.ap-northeast-1.amazonaws.com)

 When specified, it will override the region setting.
RegionAWS Region
Authentication Methodbasic
  • Uses access_key_id and secret_access_key to authenticate. See AWS Programmatic access.

    • Access Key ID

    • Secret access key

session (Recommended)
  • Uses temporary-generated access_key_id, secret_access_key and session_token.

    • Access Key ID

    • Secret access key

    • Secret token

assume_role
  • Uses role access. See AWS AssumeRole

    • TD's Instance Profile

    • Account ID

    • Your Role Name

    • External ID
    • Duration In Seconds
anonymousNot Support
Access Key IDAWS S3 issued
Secret Access KeyAWS S3 issued


Create authentication with the assume_role authentication method 

  1. Create a new authentication with the assume_role authentication method
  2. Create your AWS IAM role


Select Continue
Type a name for your connection.
Select Done.


Define your Query

  1. Complete the instructions in Creating a Destination Integration.
  2. Navigate to Data Workbench > Queries.

  3. Select a query for which you would like to export data.

  4. Run the query to validate the result set.

  5. Select Export Results.


Specify the Result Export Target

 Select Export Results.
You can select an existing authentication or create a new authentication for the external service to be used for output. Choose one of the following:

Use Existing Integration

Create a New Integration

(Optional) Specify information for Export to Amazon S3.

FieldDescription
Is user directory Root?

If selected, the user directory is treated as the root directory.

(ex. ‘/home/treasure-data’ as ‘/’)

Path prefix:The file path where the file will be stored.
Rename file after upload finishIf selected, SFTP result output renames the file on the remote SFTP server from “.xxx.tmp” to “.xxx” after all the data is transferred.
Some MA tools try to import data when a file with a specific name exists on the SFTP server. The temp name option is useful for such cases.
Format

The format of the exported files:

  • csv (comma separated)
  • tsv (tab separated) 
Compression

The compression format of the exported files:

  • None
  • GZ
  • bzip2
Header line?The header line with column name as the first line.
Delimiter

The delimited character:

  • Default
  • ,
  • Tab
  • |
Quote policy

The policy for a quote:

  • ALL
  • MINIMAL:  Add the quote character to only fields which contain delimiter, quote, or any of the characters in lineterminator.
  • NONE
Null string

How null value of the result of the query displays:

  • Default
  • empty string
  • \N
  • NULL
  • null
End-of-line character

The EOL (end-of-line) character:

  • CRLF
  • LF
  • CR
Temp filesize threshold

The maximum file size (in bytes) of a local temp file. When the temp file reaches the threshold, the file flushes to a remote file.

If you encounter the error `channel is broken`, reduce the value of this option to resolve the error.


Create an Activation Using an Integration


Integration Export Parameters for S3 

  1. Define any additional Export Results details and content review the integration parameters.
    For example, your Export Results screen might be different, or you might not have additional details to fill out.
  2. Select Done.
  3. Run your query
  4. Validate that your data moved to the destination you specified.


ParameterData TypeRequired?Supported in V1?Description
Server-side EncryptionString
yes, only sse-s3

Support values:

  • sse-s3: Server-side Encryption Mode

  • sse-kms: new SSE Mode

Server-side Encryption AlgorithmString
yes

Support value:

  • AES256 
KMS Key IDString
noSymmetric AWS KMS Key ID. If there is no input for the KMS Key ID, it will create/use the default KMS Key.
BucketStringyesyes

Provide the S3 bucket name (Ex., your_bucket_name).

PathStringyesyesSpecify the s3 filename (object key), and include an extension (Ex. test.csv).
FormatString
yesFormat of the exported file: csv, tsv, jsonl
Compression String
yesThe compression format of the exported files (Ex., None or gz)
HeaderBoolean
yesInclude a header in the exported file.
DelimiterString
yesUse to specify the delimiter character (Ex., (comma))
String for NULL valuesString
yesPlaced holder to insert for null values (Ex. Empty String)
End-of-line characterString
yesSpecify the EOL(End-Of-Line) representation (Ex. CRLF, LF)
Quote PolicyString
noUse to determine field type to quote. Support values:
  • ALL    Quote all fields
  • MINIMAL    Only quote those fields which contain delimiter, quote or any of the characters in the lineterminator.
  • NONE    Never quote fields. When the delimiter occurs in the field, escape with escape char.

Default value: MINIMAL

Quote character (Optional)Char
yesThe character used for quotes in the exported file (Ex. "). Only quote those fields which contain the delimiter, quote, or any of the characters in the lineterminator. If the input is more than 1 character, the default value will be used.
Escape character(Optional)Char
yes

The escape character is used in the exported file. If the input is more than 1 character, the default value will be used.

Part Size (MB) (Optional)Integer
no

The part size in multipart upload.

Default 10, min 5, max 5000

Example Query

SELECT * FROM www_access


(Optional) Schedule Query Export Jobs

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.


(Optional) Configure Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Exporting Data with Parameters.

S3 (v2) Configuration Keys

NameTypeRequiredDescription
bucketStringYes
pathStringYes
sse_typeString
sse-s3, sse-kms

sse_algorithm
String
AES256
kms_key_id
String

formatString
csv, tsv, jsonl
compressionString
none, gz
headerBoolean
Default true
delimiterString
default , \t |
null_valueString
default, empty, \\N, NULL, null
newlineString
CR, LF, CRLF
quote_policyString
ALL, MINIMAL, NONE
escapeChar

quoteChar

part_sizeInteger

Example Workflow for S3 (v2)

_export:
  td:
  database: td.database

+s3v2_test_export_task:
  td>: export_s3v2_test.sql
  database: ${td.database}
  result_connection: s3v2_conn
  result_settings:
  	bucket: my-bucket
  	path: /path/to/target.csv
  	sse_type: sse-s3
  	format: csv
  	compression: gz
  	header: false
    delimiter: default
    null_value:  empty
    newline: LF
  	quote_policy: MINIMAL
  	escape: '"'
  	quote: '"'
  	part_size: 20

(Optional) Export Integration Using the CLI

To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3.
You can specify detailed settings to export your S3 via the --result parameter. 

Creating authentication with Assume Role is only supported through the console. Attempting to create it through the TD CLI will result in an error.

Example CLI Command for S3 (v2)

td query \
--result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto