Learn more about KARTE Import Integration.
You can use the Karte connector to write job results from queries run in Treasure Data to your Google Cloud Storage.
Prerequisites
Basic knowledge of Treasure Data, including the Toolbelt.
A Google Cloud Platform account
Getting Google Cloud Platform Credentials
Before creating the connector, you need to gather the following information: your Project ID, and JSON Credential.
JSON Credential
The integration with Google Cloud Storage is based on server-to-server API authentication. Go to your Google Developer Console and select Credentials under APIs & auth at the left menu. Then, select Service account.
Select the JSON based key type, which is Google’s recommended configuration. The key is automatically downloaded by the browser.
The Service Account owner, who generated JSON Credentials, must have Write permission for destination bucket. |
Specifying Output in Your KARTE Connector
Using the TD Console
Create the destination Bucket in Google Cloud Storage
Create your bucket from your Google Cloud Storage console.
Write the query
Go to the Treasure Data Console query editor page and compile your query.
Specify the Result Export target
In the same window, select the Create New Connection in the Result Export.
In the dialog opens, search for and select your connection type: KARTE. Provide a name for your KARTE connection. Complete the information, including your JSON Credential, Bucket name, and Path:
Execute the query
Finally, either save the query with a name or just run the query. After the successful completion of the query, the results are automatically sent to the specified Google Cloud Storage destination.
Use the CLI
You can set a scheduled query to send job results to Google Cloud Storage using KARTE.
Designate your json_key and escape the newline with backslash. |
Example
$ td sched:create scheduled_karte "10 6 * * *" \ -d dataconnector_db "SELECT id,account,purchase,comment,time FROM data_connectors" \ -r '{"type":"karte","bucket":"samplebucket","path_prefix":"/output/test.csv","format":"csv","compression":"","header_line":false,"delimiter":",","null_string":"","newline":"CRLF", "json_keyfile":"{\"private_key_id\": \"ABCDEFGHIJ\", \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nABCDEFGHIJ\\ABCDEFGHIJ\\n-----END PRIVATE KEY-----\\n\", \"client_email\": \"ABCDEFGHIJ@developer.gserviceaccount.com\", \"client_id\": \"ABCDEFGHIJ.apps.googleusercontent.com\", \"type\": \"service_account\"}"}'
(Optional) Configure Export Results in Workflow
Within Treasure Workflow, you can specify the use of this data connector to output data.
timezone: UTC _export: td: database: sample_datasets +td-result-into-target: td>: queries/sample.sql result_connection: karte_integration result_settings: format: csv newline: CR ....
For more information on using data connectors in Treasure Data Workflow to export data, review About Using Workflows to Export Data with TD Toolbelt.
KARTE Connector Parameters
You can use the following parameters when configuring the KARTE connector.
format | "csv" or "tsv" |
---|---|
compression | "" or "gz" |
header_line | true or false |
delimiter | "," or "\t" or "|" |
null_string | "" or "\N" |
newline | "CRLF" or "CR" or "LF" |
json_keyfile | Use backslash to escape newline '\n" |