Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Create buckets: Create and name a bucket that stores data.
  • Storing data: Store an infinite amount of data in a bucket.

Differences between Amazon S3 Export Integration v2 and Amazon S3 Export Integration v1

Review information in the following table to understand the differences and potential advantages between v2 and v1.

FeatureAmazon S3 v2Amazon S3 v1
Server-side Encryption with Customer Master Key (CMK)
stored in AWS Key Management Service
X
Support for Quote Policy for output data formatX
Support Assume Role authentication methodX


This topic includes:

Table of Contents
maxLevel3
excludeWhat can you do with this Integration?

...

  • if not input KMS key id, it will create/using the default KMS key

  • if input KMS Key  ID, you must choose asymmetric symmetric CMK, not asymmetric CMKs

  • The AWS KMS CMK must be in the same Region as the bucket

...

Numbered Headings
start-numbering-with1
start-numbering-ath5
Open TD Console.
Navigate to Integrations Hub Catalog.
Search for S3 and select
Image Removed
AmazonS3.
Select Create Authentication.


Image Added
Type the credentials to authenticate:
ParameterDescription

Endpoint

S3 service endpoint override. You can find region and endpoint information from AWS Document. (Ex. s3-.ap-northeast-1.amazonaws.com)

 When specified will override region setting
RegionAWS Region
Authentication Methodbasic
  • Uses access_key_id and secret_access_key to authenticate. See AWS Programmatic access.

    • Access Key ID

    • Secret access key

session (Recommended)
  • Uses temporary-generated access_key_id, secret_access_key and session_token.

    • Access Key ID

    • Secret access key

    • Secret token

assume_role
  • Uses role access. See AWS AssumeRole

    • TD's Instance Profile

    • Account ID

    • Your Role Name

    • External ID
    • Duration In Seconds
anonymousNot Support
Access Key IDAWS S3 issued
Secret Access KeyAWS S3 issued


Image RemovedImage Added

Info

Create authentication with the assume_role authentication method 

  1. Create a new authentication with the assume_role authentication method
    Image Added
  2. Create your AWS IAM role
    Image AddedImage Added

Select Continue
Type a name for your connection.
Select Done.


Define your Query

Numbered Headings
  1. Complete the instructions in Creating a Destination Integration.
  2. Navigate to Data Workbench > Queries.

  3. Select a query for which you would like to export data.

  4. Run the query to validate the result set.

  5. Select Export Results.

Image Modified

  1. Select an existing integration authentication.
  2. Define any additional Export Results details. In your export integration content review the integration parameters.
    For example, your Export Results screen might be different, or you might not have additional details to fill out:

  3. Select Done.
  4. Run your query
  5. Validate that your data moved to the destination you specified.

...


Integration Export Parameters for S3 


ParameterData TypeRequired?Supported in V1?Description
Server-side EncryptionString
yes, only sse-s3

Support values:

  • sse-s3: Server-side Encryption Mode

  • sse-kms: new SSE Mode

Server-side Encryption AlgorithmString
yes

Support value:

  • SEA256 
KMS Key IDString
noSymmetric AWS KMS Key Id, if not input KMS key id, it will create/using the default KMS key
BucketStringyesyes

Provide the S3 bucket name (Ex. your_bucket_name)

PathStringyesyesSpecify s3 filename (object key), include an extension (Ex. test.csv)
FormatString
yesFormat of the exported file: csv, tsv, jsonl
Compression String
yesThe compression format of the exported files (Ex. None or gz)
HeaderBoolean
yesInclude header in the exported file
DelimiterString
yesUse to specify the delimiter character (Ex, (comma))
String for NULL valuesString
yesPlaced holder to insert for null values (Ex. Empty String)
End-of-line characterString
yes Specify the EOL(End-Of-Line) representation (Ex. CRLF, LF)
Quote PolicyString
noUse to determine field type to quote. Support values:
  • ALL    Quote all fields
  • MINIMAL    Only quote those fields which contain delimiter, quote or any of the characters in lineterminator
  • NONE    Never quote fields. When the delimiter occurs in field, escape with escape char

Default value: MINIMAL

Quote character (Optional)Char
yesThe character used for quotes in the exported file(Ex. "). Only quote those fields which contain delimiter, quote, or any of the characters in lineterminator. If the input is more than 1 character, the default value will be used
Escape character(Optional)Char
yes

The escape character is used in the exported file. If the input is more than 1 character, the default value will be used

Part Size (MB) (Optional)Integer
no

The part size in multipart upload 

Default 10, min 5, max 5000

...

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Using Workflows to Export Exporting Data with the TD ToolbeltParameters.

S3 (v2) Configuration Keys

...

To output the result of a single query to an S3 buck add the --result option to the td query command. After the job is finished, the results are written into your s3
You can specify detailed settings to export your S3 via --result parameter. 

Only support create authentication with Assume Role using the console, creating through TD CLI will result in an error

Example for CLI command for S3 (v2)

...