Learn more about Repro Import Integration.
You can use the new Repro Export Integration connector to export files to your Repro’s Amazon S3 buckets with customized parameters for an easy configuration.
Prerequisites
Basic knowledge of Treasure Data, including the TD Toolbelt.
An S3 bucket with region ID.
Using the TD Console to Create Your Connection
Create a New Connection
When you create a data connection, you must provide authentication to access the integration. In Treasure Data, you configure the authentication first and then specify the source information.
Open TD Console.
Navigate to Integrations Hub > Catalog
Search for and select Repro.
The following dialog opens:
Enter a name for your connection and click Done
Configure Export Results in Your Data Connection
In this step you create or reuse a query. In the query, you configure the data connection. You may need to define the column mapping in the query.
Configure the Connection by Specifying the Parameters
Open the TD console.
Navigate to Data Workbench > Queries.
Select the query that you plan to use to export data.
Click Export Results located at top of your query editor. The Choose Integration dialog opens. You have two options when selecting a connection to use to export the results, using an existing connection or creating a new one.
Use an existing connection
Type the connection name in the search box to filter.
Select your connection.
Create a new Repro Connection.
Fill in the field values to create the new connection.
Enter the required credentials for your new connection. Set the following parameters.
Parameter | Description |
---|---|
Use AWS S3 Server-Side Encryption (optional) | Use S3 Server-side encryption |
Server-Side Encryption algorithm: (optional) | The algorithm used for encryption |
Bucket (required) | The bucket name in s3 |
File Path (required) | The full path of the file includes file name and extension, i.e: production/<app_id>/user-list/filename.csv.gz |
Format (required) | File export to csv, required extension in the file path |
Header line (required) | The export file will contain first row as columns name |
Null String (optional) | The value that will replace null in the file |
End-of-line character (optional) | Character marked as end-of-line in the file |
Quote Policy (required) | Quote policy for the file |
Compression (required) | Compress file in gz, required extension in the file path |
Here is a sample configuration:
Example of a Query to Populate Repro
From Treasure Data, run the following query with export results into a connection for Repro:
Code Example
SELECT an_email_column AS EMAIL, another_phone_column AS PHONE FROM your_table;
Optional: Use of Scheduled Jobs for Export
You can use Scheduled Jobs with Result Export, to periodically write the output result to a target destination that you specify.
Optional: Configure Export Results in Workflow
Within Treasure Workflow, you can specify the use of this data connector to export data.
timezone: UTC _export: td: database: sample_datasets +td-result-into-target: td>: queries/sample.sql result_connection: your_connections_name result_settings: type: repro bucket: bucket_name region: ap-northeast-2 use_sse: true sse_algorithm: AES256 auth_method: basic session_token: session_token path: /td-export-repro/file_output.csv access_key_id: access_id secret_access_key: secret_key formatter: {type: csv, delimiter: "\t", newline: CRLF, newline_in_field: LF, charset: UTF-8, quote_policy: MINIMAL, quote: '"', escape: \, null_string: \N, default_timezone: UTC} encoders: {type: gzip}
Click here for more information on using data connectors in workflow to export data.