# Treasure Data Data Exchange Export Integration You can use the Treasure Data connector to write job results to a Treasure Data database. Usually, this connector is used to export data from one Treasure Data account to another Treasure Data account. For sample workflows on writing job results from one TD account to another TD account, view [Treasure Boxes](https://github.com/treasure-data/treasure-boxes/tree/master/td/another_td). We strongly recommend that you not use this connector to write data from one table in your account to another table in your account. Instead, use "INSERT INTO" query or "INSERT OVERWRITE/CREATE TABLE AS SELECT" query for the best performance. Presto: "CREATE TABLE AS SELECT" or "INSERT INTO" Hive: "INSERT INTO/OVERWRITE" ## Prerequisites - Basic knowledge of Treasure Data, including the TD [Toolbelt](https://toolbelt.treasuredata.com/). ## Basic Usage You can configure a connection from TD Console or from the CLI. ## Create a Connection in TD Console You can configure a connection from within TD Console. To complete an account-to-account transfer, begin from the target Treasure Data account. You must copy the API Key information from the target account profile. ### Retrieve the API Key Information 1. Navigate to the **TD Console** > **My Settings.** You must have admin permissions. ![](/assets/image-20200513-172241.5c1fd8360e6e917db074642a3c64e44eebb8437b573de805f2af6aab6c90ba6c.e4e4f55a.png) 2. Select **API Keys**. ![](/assets/image-20200513-173834.f0a0fc83c7c560d2b61b90a7aaf744608ec66e1f1fbfbaece53dbcd901f601b8.e4e4f55a.png) 3. Validate access, if needed. ![](/assets/image-20200513-174119.f2995567ec7192e1ab7645862385a7a8fe781aa209d90f43e7e627c6e199047b.e4e4f55a.png) 4. When validation is complete and you return to the profile, and you can see both a master key and write-only key. Highlight the Write-only API key. By selecting it, it automatically copies to your clipboard. ![](/assets/image-20200513-175122.921f399aea98b7b9799edd4a696979114197fa1a4f0d412398cba5c67e70d4f6.e4e4f55a.png)![](/assets/image-20200513-174543.be19e382e9fe1cb13dd26b549030fecc822723c49fdf29b22685cdb03e4c41d4.e4e4f55a.png) If your data transfer involves updating the existing data, e.g. with mode=replace, you'll need to use the master key instead of the write-only keyy. ### Create a New Authentication When you configure a data connection, you provide authentication to access the integration. In Treasure Data, you configure the authentication and then specify the source information. 1. Open **TD Console.** 2. Navigate to **Integrations Hub** > **Catalog**. 3. Search and select the Treasure Data tile. ![](/assets/image-20200513-180817.290d16ad023e623f4ac349d5101008c4111197c95ea0d544bd9c79869ca49733.e4e4f55a.png) 4. Select **Create**. Enter the write-only **API Key** from the target account. In **API Hostname** enter the server to use for export. You can leave the field blank to use the default server for your region. Alternatively, you can enter one of the following values: - US: [*api.treasuredata.com*](http://api.treasuredata.com) - JP: [*api.treasuredata.co.jp*](http://api.treasuredata.co.jp) - EU01: [*api.eu01.treasuredata.com*](http://api.eu01.treasuredata.com) - AP02 (Korea): [*api.api02.treasuredata.com*](http://api.api02.treasuredata.com) ![](/assets/image-20200513-181011.db17ca82046db3cfa4bac40094e86d1336f22cde90ac6f6a70fb7e9e9f628944.e4e4f55a.png) 1. Select **Continue.** Name your connection and **select Create Connection.** If you are writing data from one region to another, e.g. from AWS US to AWS JP the API Hostname field is required. ### Specify Output in your Treasure Data Connector To create a transfer, create or reuse a query that configures the data connection. 1. Open the **TD Console**. 2. Navigate to the Data Workbench > Queries. Highlight the query that you plan to use to export data. 3. Select **Export Results** in the top right of the query editor. ![](/assets/image-20200513-203249.de106fdc17b785c33dca5a00b00d63e66453cd0b3fb074638715ead7768f4400.e4e4f55a.png) ```sql SELECT code, COUNT(1) FROM www_access GROUP BY code ``` 4. The **Choose Integration** dialog opens. ![](/assets/image-20200513-202807.858b6fbbfd044c51a986cc90474cf386233b53729ac171ff1189171d158b4cf1.e4e4f55a.png) 5. Select **Use Existing Integration** and search for the name of the saved authentication. ### Specifying the Parameters 1. After selecting the authentication, select **Next**. The **Export Results** dialog opens. ![](/assets/treasuredatadde.b223eeb60dde65c060f8a57a544233ba864ea1e02fb197c3e34507cbe90bfe03.e4e4f55a.png) 2. Specify the parameters and select **Done**. | Parameters | Description | | --- | --- | | Database name (required) | The name of an existing database. | | Table name (required) | The table name. A new table will be created if the table does not exist. | | Mode (required) | Choose the Append or Replace method for importing the data.- **Append** (default)-Data import results are appended to the table. If the table does not exist, it will be created. - **Replace**-Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created. | | Time (optional) | User-defined time value. It is applied if there is no time column in the query result. | | File split size (KB) | Use to split file when file size is larger or equal to this value (16MB-512MB) | ### (Optional) Schedule Query Export Jobs You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify. Treasure Data's scheduler feature supports periodic query execution to achieve high availability. When two specifications provide conflicting schedule specifications, the specification requesting to execute more often is followed while the other schedule specification is ignored. For example, if the cron schedule is `'0 0 1 * 1'`, then the 'day of month' specification and 'day of week' are discordant because the former specification requires it to run every first day of each month at midnight (00:00), while the latter specification requires it to run every Monday at midnight (00:00). The latter specification is followed. #### Scheduling your Job Using TD Console 1. Navigate to **Data Workbench > Queries** 2. Create a new query or select an existing query. 3. Next to **Schedule**, select None. ![](/assets/image2021-1-15_17-28-51.f1b242f6ecc7666a0097fdf37edd1682786ec11ef80eff68c66f091bc405c371.0f87d8d4.png) 4. In the drop-down, select one of the following schedule options: ![](/assets/image2021-1-15_17-29-47.45289a1c99256f125f4d887e501e204ed61f02223fde0927af5f425a89ace0c0.0f87d8d4.png) | Drop-down Value | Description | | --- | --- | | Custom cron... | Review [Custom cron... details](#custom-cron-details). | | @daily (midnight) | Run once a day at midnight (00:00 am) in the specified time zone. | | @hourly (:00) | Run every hour at 00 minutes. | | None | No schedule. | #### Custom cron... Details ![](/assets/image2021-1-15_17-30-23.0f94a8aa5f75ea03e3fec0c25b0640cd59ee48d1804a83701e5f2372deae466c.0f87d8d4.png) | **Cron Value** | **Description** | | --- | --- | | `0 * * * *` | Run once an hour. | | `0 0 * * *` | Run once a day at midnight. | | `0 0 1 * *` | Run once a month at midnight on the morning of the first day of the month. | | "" | Create a job that has no scheduled run time. | ``` * * * * * - - - - - | | | | | | | | | +----- day of week (0 - 6) (Sunday=0) | | | +---------- month (1 - 12) | | +--------------- day of month (1 - 31) | +-------------------- hour (0 - 23) +------------------------- min (0 - 59) ``` The following named entries can be used: - Day of Week: sun, mon, tue, wed, thu, fri, sat. - Month: jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec. A single space is required between each field. The values for each field can be composed of: | Field Value | Example | Example Description | | --- | --- | --- | | A single value, within the limits displayed above for each field. | | | | A wildcard `'*'` to indicate no restriction based on the field. | `'0 0 1 * *'` | Configures the schedule to run at midnight (00:00) on the first day of each month. | | A range `'2-5'`, indicating the range of accepted values for the field. | `'0 0 1-10 * *'` | Configures the schedule to run at midnight (00:00) on the first 10 days of each month. | | A list of comma-separated values `'2,3,4,5'`, indicating the list of accepted values for the field. | `0 0 1,11,21 * *'` | Configures the schedule to run at midnight (00:00) every 1st, 11th, and 21st day of each month. | | A periodicity indicator `'*/5'` to express how often based on the field's valid range of values a schedule is allowed to run. | `'30 */2 1 * *'` | Configures the schedule to run on the 1st of every month, every 2 hours starting at 00:30. `'0 0 */5 * *'` configures the schedule to run at midnight (00:00) every 5 days starting on the 5th of each month. | | A comma-separated list of any of the above except the `'*'` wildcard is also supported `'2,*/5,8-10'`. | `'0 0 5,*/10,25 * *'` | Configures the schedule to run at midnight (00:00) every 5th, 10th, 20th, and 25th day of each month. | 1. (Optional) You can delay the start time of a query by enabling the Delay execution. ### Execute the Query Save the query with a name and run, or just run the query. Upon successful completion of the query, the query result is automatically exported to the specified destination. Scheduled jobs that continuously fail due to configuration errors may be disabled on the system side after several notifications. (Optional) You can delay the start time of a query by enabling the Delay execution. ## Activate a Segment in Audience Studio You can also send segment data to the target platform by creating an activation in the Audience Studio. 1. Navigate to **Audience Studio**. 2. Select a parent segment. 3. Open the target segment, right-mouse click, and then select **Create Activation.** 4. In the **Details** panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters. 5. Customize the activation output in the **Output Mapping** panel. ![](/assets/ouput.b2c7f1d909c4f98ed10f5300df858a4b19f71a3b0834df952f5fb24018a5ea78.8ebdf569.png) - Attribute Columns - Select **Export All Columns** to export all columns without making any changes. - Select **+ Add Columns** to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select **+ Add Columns**to add new columns for your activation output. - String Builder - **+ Add string** to create strings for export. Select from the following values: - String: Choose any value; use text to create a custom value. - Timestamp: The date and time of the export. - Segment Id: The segment ID number. - Segment Name: The segment name. - Audience Id: The parent segment number. 1. Set a **Schedule**. ![](/assets/snippet-output-connector-on-audience-studio-2024-08-28.a99525173709da1eb537f839019fa7876ffae95045154c8f2941b030022f792c.8ebdf569.png) - Select the values to define your schedule and optionally include email notifications. 1. Select **Create**. If you need to create an activation for a batch journey, review [Creating a Batch Journey Activation](/products/customer-data-platform/journey-orchestration/batch/creating-a-batch-journey-activation). ## For On-Demand Jobs from the CLI For on-demand jobs, just add the `--result` option to the `td query` command. When the job is finished, the results are written into `output_table` within the `testdb` database. ```bash $ td query --result 'td://@/testdb/output_table' \ -w -d testdb \ "SELECT code, COUNT(1) FROM www_access GROUP BY code" ``` ### For Scheduled Jobs For scheduled jobs, just add the `--result` option when scheduling a job. Every time the job runs, the results are written into `output_table`. ```bash $ td result:create mydb td://@/testdb $ td sched:create hourly_count_example "0 * * * *" \ -d testdb "select count(*) from www_access" \ --result mydb:output_table ``` ### Two Ways to Modify Data You can add or delete data in two ways: - append - replace ``` td://@/testdb/output_table # append td://@/testdb/output_table?mode=append # append td://@/testdb/output_table?mode=replace # replace ``` #### Append This is the default mode. The query results are appended to the table. If the table does not exist yet, a new table will be created. This operation is not idempotent; you cannot make that same call repeatedly and produce the same result. This mode is not [atomic](https://docs.treasuredata.com/smart/project-product-documentation/glossary). #### Replace If the table already exists, the rows of the existing table are replaced with the query results. If the table does not exist yet, a new table will be created. We achieve [**atomicity**](https://docs.treasuredata.com/smart/project-product-documentation/glossary) (so that a consumer of the table always has consistent data) by performing the following three steps in a **single transaction**. 1. Create a temporary table. 2. Write to the temporary table. 3. Replace the existing table with the temporary table atomically.