This guide is to explain a new feature called "Reuse Connector Authentication." This feature allows clients to reuse an existing connector authentication
Instead of creating a new authentication for each new integration, you can reuse existing authentications. This is more secure and saves time when setting up the integration. It also makes integrations that only support OAuth2 authentication method available for workflow and the TD Toolbelt CLI.
If required, create a new authentication:
- Open the TD Console.
- Navigate to Integrations Hub > Catalog.
- Search for your Integration in the Catalog; then hover your mouse over the icon and select Create Authentication.
- Ensure that the Credentials tab is selected and then enter credential information for the integration.
Authentication details vary depending on the type see About Treasure Data Integrations. Here is an example authentication page:
- Select Continue.
- Enter a name for your authentication and select Done.
To reuse an existing authentication, you first need to identify the Authentication ID. This ID is usually displayed in the browser URL of the Authentications. Following the step here to get the Authentication ID of a saved authentication.
- On TD Console UI.
- Navigate to Integrations Hub > Authentications
- Click on a saved authentication.
- Copy and save the Authentication ID in the browser URL.
To reuse an existing authentication, specify the config key td_authentication_id with the value of the Authentication ID. In the example above, the value would be 255574.
This config key can be used in the following CLI commands:
- connector:guess
- connector:preview
- connector:issue
- connector:create
- connector:update
- sched:create
- sched:update
- query
- export:result
It can also be used with the following workflow operators:
- td
- td_run
- td_load
For new job or workflow configurations, simply set the td_authentication_id to the Authentication ID. In some cases, you will need to first create a config.yml file the has been configured with the specific td_authentication_id.
For example, here is an S3_v2 integration that uses the authentication method assume_role.
- Configuration file for connector:guess
In this file, named s3_v2_seed.yml, the td_authentication_id is set to the authentication ID 260370. This file will be used to create the config.yml file.
---
in:
type: s3_v2
td_authentication_id: 260370
bucket: qa-auto-test-3
path_prefix: 'qh_s3connector_test/Compress/assume_role_staging.csv'- connector:guess results
connector:guess uses the s3_v2_seed.yml to create the config.yml file which is used by TD Toolbelt CLI commands.
td -k 1/xxx \
-e https://api.treasuredata.com \
connector:guess s3_v2_seed.ymlGuessed configuration:
---
in:
type: s3_v2
bucket: qa-auto-test-3
path_prefix: qh_s3connector_test/Compress/assume_role_staging.csv
parser:
charset: UTF-8
newline: CRLF
type: csv
delimiter: ","
quote: "\""
escape: "\""
trim_if_not_quoted: false
skip_header_lines: 1
allow_extra_columns: false
allow_optional_columns: false
columns:
- {name: path, type: string}
- {name: cnt, type: string}
- {name: user, type: string}
- {name: host, type: string}
- {name: referer, type: string}
- {name: code, type: long}
- {name: agent, type: string}
- {name: size, type: long}
- {name: method, type: string}
- {name: time, type: long}
td_authentication_id: 260370
out: {}
exec: {}
filters: []Created config.yml file. Use 'td -k 1/xxx -e https://console.treasuredata.com connector:preview config.yml' to see bulk load preview.
- connector:preview results
td -k 1/xxx \
-e https://console.treasuredata.com \
connector:preview config.yml- connector:issue
td -k 1/xxx \
-e https://api.treasuredata.com \
connector:issue config.yml --database tuan_db --table test_tbl- connector:create
td -k 1/xxxx \
-e https://api.treasuredata.com \
connector:create bulkload_data_from_s3 \
"0 * * * *" tuan_db test_tbl config.yml- connector:update
td -k 1/xxx -e https://api.treasuredata.com connector:update bulkload_data_from_s3 -c config.yml -s '0 0 * * *'- sched:create
td -k 1/xxx \
-e https://api.treasuredata.com \
sched:create result_to_s3_v2 '0 0 * * *' \
--database tuan_db \
'SELECT * from aaks' \
--result '{"type":"s3_v2","td_authentication_id":260370,"duration_in_seconds":3600,"bucket":"qa-auto-test-3","path":"qh_s3connector_test/Compress/assume_role_staging1.csv","format":"csv","compression":"none","header":true}'- sched:update
td -k 1/xxx \
-e https://api.treasuredata.com \
sched:update result_to_s3_v2 -s '0 0 * * *' \
--database tuan_db \
--result '{"type":"s3_v2","td_authentication_id":260370,"duration_in_seconds":3600,"bucket":"qa-auto-test-3","path":"qh_s3connector_test/Compress/assume_role_staging2.csv","format":"csv","compression":"none","header":true}'- query
td -k "1/xxxx" \
-e https://api.treasuredata.com \
query --database tuan_db \
--wait "SELECT * from aaks" --type presto \
--result '{"type":"s3_v2","td_authentication_id":260370,"duration_in_seconds":3600,"bucket":"qa-auto-test-3","path":"qh_s3connector_test/Compress/assume_role_staging.csv","format":"csv","compression":"none","header":true}'- export:result
td -k "1/xxxx" \
-e https://api.treasuredata.com \
export:result 1696822041 \
'{"type":"s3_v2", "td_authentication_id":260370, "bucket": "qa-auto-test-us-east-1", "path":"quang/test_export_result_02.csv"}'- td_load
In this example, the td_load command uses the s3_v2_guessed.yml where td_authentication_id is set to 260370..
---
in:
type: s3_v2
td_authentication_id: 260370
bucket: qa-auto-test-3
path_prefix: 111time.csv
parser:
charset: UTF-8
newline: CRLF
type: csv
delimiter: ","
quote: "\""
escape: "\""
trim_if_not_quoted: false
skip_header_lines: 1
allow_extra_columns: false
allow_optional_columns: false
columns:
- name: col_1
type: string
- name: col_2
type: string
- name: col_3+import_from_s3_assume_role_with_existing_connection:
td_load>: s3_v2_guessed.yml
database: tuan_data
table: wf_test- query
+query_data_and_write_result_with_existing_connection_by_url:
td>:
query: 'select * from wf_test'
database: tuan_data
result_url: '{"td_authentication_id": 260370, "bucket": "qa-auto-test-3", "path": "viet/assume_role_from_eu01.csv"}'- td_run
In this example, presto_select_all_s3v2 is a saved query.
+query_sql:
td_run>: presto_select_all_s3v2
database: quang
result_url: '{"type":"s3_v2", "td_authentication_id":260370, "bucket": "qa-auto-test-us-east-1", "path":"quang/wf_jan16_1444.csv"}'- td_result_export
This operator supports the reuse of an existing integration authentication in the legacy manner, by returning the name of the authentication on TD console UI in result_connection.
_export:
td:
database: quang
+export_query_result:
td_result_export>:
job_id: 14660140
result_connection: quang_s3v2_happy_case_1195
result_settings:
bucket: qa-auto-test-us-east-1
path: quang/s3v2_wf_jan16_1655.csvFor existing job and workflow configurations, simply remove the existing connector authentication declarations from the configuration, and replace them with td_authentication_id specifying the authentication ID. In this example, the authentication that was obtained from the end of the authentication URL in TD Console is 11.
