Amazon S3 Export Integration V2

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers scalability, data availability, security, and performance. Amazon S3 provides features for data organization and configuration of access controls for your business, organization, and compliance requirements.

This TD export integration allows you to write job results from Treasure Data directly to Amazon S3.

What can you do with this Integration?

• Store data: Store an infinite amount of data in a bucket.

Differences between Amazon S3 Export Integration v2 and v1

Review the information in the following table to understand the features that are available in v2 that are not available in v1.

Prerequisites

• The following AWS permissions for the IAM User :
  • with s3:PutObject, s3:AbortMultipartUpload permissions.
  • with kms:Decrypt, kms:GenerateDataKey permissions when selecting the sse-kms setting.
• (Optional) Basic knowledge of Treasure Data, including the TD Toolbelt.

Requirements and Limitations

• The default query result limit for export to S3 is 100GB. You can configure the part size setting up to 5000 (MB). The file limit will be about 5TB.
• The default export format is CSV RFC 4180.
• Output in TSV and JSONL format are also supported.
• TLS v.1.2 supported

Static IP Address of Treasure Data Integration

If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.

Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/

About S3 Server-Side Encryption

You can encrypt upload data with AWS S3 Server-Side Encryption. You don't need to prepare an encryption key. Data will be encrypted at the server side with 256-bit Advanced Encryption Standard (AES-256).

Use the Server-Side Encryption bucket policy if you require server-side encryption for all objects that are stored in your bucket. When you have the server-side encryption enabled, you don't have to turn on the SSE option. However, job results may fail if you have bucket policies that reject HTTP requests without encryption information.

About KMS Server-Side Encryption

You can encrypt upload data with Amazon S3-managed encryption keys (SSE-S3).

When you enable AWS KMS for server-side encryption in Amazon S3:

• If you don't input the KMS Key ID, it will use (or create) the default KMS key.
• If you input the KMS Key ID, you must choose symmetric CMK (not asymmetric CMKs).
• The AWS KMS CMK must be in the same region as the bucket.

Using the TD Console to Create a Connection

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

Create a New Authentication

1. Open TD Console.
2. Navigate to Integrations Hub > Catalog.
3. Search for S3 v2and select Amazon S3 (v2).
4. Select Create Authentication.

A new Authentication dialog opens. Depending on the Authentication method you choose, the dialog may look like one of these screens:

1. Configure the authentication fields, and then select Continue.

2. Name your new AWS S3 connection, and select Done.

Creating an Authentication with the assume_role authentication method

1. Create a new authentication with the assume_role authentication method.
2. Make a note of the numeric portion of the value in the TD's Instance Profile field.

1. Create your AWS IAM role.

Defining your Query

1. Select Export Results.
2. You can select an existing authentication or create a new authentication for the external service to be used for output. Choose one of the following:

• Use Existing Integration

• Create a New Integration

(Optional) Specify information for Export to Amazon S3.

Defining Integration Export Parameters for S3

1. Define any additional Export Results details and content review the integration parameters. For example, your Export Results screen might be different, or you might not have additional details to fill out.

1. Select Done.
2. Run your query
3. Validate that your data moved to the destination you specified.

Example Query

SELECT * FROM www_access

(Optional) Schedule Query Export Jobs

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

Treasure Data's scheduler feature supports periodic query execution to achieve high availability.

When two specifications provide conflicting schedule specifications, the specification requesting to execute more often is followed while the other schedule specification is ignored.

For example, if the cron schedule is '0 0 1 * 1', then the 'day of month' specification and 'day of week' are discordant because the former specification requires it to run every first day of each month at midnight (00:00), while the latter specification requires it to run every Monday at midnight (00:00). The latter specification is followed.

Scheduling your Job Using TD Console

1. Navigate to Data Workbench > Queries

2. Create a new query or select an existing query.

3. Next to Schedule, select None.

4. In the drop-down, select one of the following schedule options:

   Drop-down Value	Description
   Custom cron...	Review Custom cron... details.
   @daily (midnight)	Run once a day at midnight (00:00 am) in the specified time zone.
   @hourly (:00)	Run every hour at 00 minutes.
   None	No schedule.

Custom cron... Details

 *    *    *    *    *
 -    -    -    -    -
 |    |    |    |    |
 |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 |    |    |    +---------- month (1 - 12)
 |    |    +--------------- day of month (1 - 31)
 |    +-------------------- hour (0 - 23)
 +------------------------- min (0 - 59)

The following named entries can be used:

• Day of Week: sun, mon, tue, wed, thu, fri, sat.
• Month: jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec.

A single space is required between each field. The values for each field can be composed of:

1. (Optional) You can delay the start time of a query by enabling the Delay execution.

Execute the Query

Save the query with a name and run, or just run the query. Upon successful completion of the query, the query result is automatically exported to the specified destination.

(Optional) You can delay the start time of a query by enabling the Delay execution.

(Optional) Configuring Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Exporting Data with Parameters.

S3 (v2) Configuration Keys

Example Workflow for S3 (v2)

_export:
  td:
  database: td.database

+s3v2_test_export_task:
  td>: export_s3v2_test.sql
  database: ${td.database}
  result_connection: s3v2_conn
  result_settings:
    bucket: my-bucket
    path: /path/to/target.csv
    sse_type: sse-s3
    format: csv
    compression: gz
    header: false
    delimiter: default
    null_value:  empty
    newline: LF
    quote_policy: MINIMAL
    escape: '"'
    quote: '"'
    part_size: 20
    json_columns: col_map,col_array,col_json_string

(Optional) Exporting Integration Using the CLI

To output the result of a single query to an S3 bucket add the "--result option" to the td query command. After the job is finished, the results are written into your S3 bucket.

You can specify detailed settings to export your S3 via the "--result parameter".

Create an Authentication with Assume Role is only supported through the console UI. Follows the instruction Reuse the existing Authentication to get the Authentication ID, and reuse it for CLI

Example CLI Command for S3 (v2)

td query \
--result '{"type":"s3_v2","auth_method":"basic","region":"us-east-2","access_key_id": "************","secret_access_key":"***************","bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto

Example CLI command with an existing Authentication ID

td query \
--result '{"type":"s3_v2","td_authentication_id": 77348,"bucket":"bucket_name","path":"path/to/file.csv","format":"csv","compression":"none","header":true,"delimiter":"default","null_value":"default","newline":"CRLF","quote_policy":"NONE","part_size":10}' \
-w -d testdb \
"SELECT 1 as col" -T presto