Writing Job Results into Google Cloud Storage

This article explains how to write job results directly to your Google Cloud Storage.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data, including the toolbelt.
  • A Google Cloud Platform account

Get the Google Cloud Platform credentials

To use this feature, you need the following information: your Project ID, and JSON Credential.

JSON Credential

The integration with Google Cloud Storage is based on server-to-server API authentication. First, please visit your Google Developer Console and click Credentials under APIs & auth at the left menu. Then, select Service account:



Please select the JSON based key type which is Google’s recommended configuration. The key will be automatically downloaded by the browser.



Untitled-3
Service Account who generated JSON Credentials has to have Write permission for destination bucket.

Usage

Use from Web Console

Create the destination Bucket in Google Cloud Storage

First, please create your bucket from your Google Cloud Storage console.

Write the query

Please visit the Treasure Data Console query editor page and compile your query.

Specify the Result Export target

In the same window, click the Add button in the Result Export section and select Google Cloud Storage from the drop down menu. Please fill out all the information below, including your JSON Credential, Bucket name, and Path:



Execute the query

Finally, either save the query with a name and run or just run the query. Upon successful completion of the query, its result will be automatically imported to the specified Google Cloud Storage destination:



Use from CLI

The following command is allow to set a sheduled query with Result Output to Google Cloud Storage.

Untitled-3
Please designate your json_key and escape newline with backslash.

For example,

$ td sched:create scheduled_gcs "10 6 * * *" \
-d dataconnector_db "SELECT id,account,purchase,comment,time FROM data_connectors" \
-r '{"type":"gcs","bucket":"samplebucket","path_prefix":"/output/test.csv","format":"csv","compression":"","header_line":false,"delimiter":",","null_string":"","newline":"CRLF",  "json_keyfile":"{\"private_key_id\": \"ABCDEFGHIJ\", \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nABCDEFGHIJ\\ABCDEFGHIJ\\n-----END PRIVATE KEY-----\\n\", \"client_email\": \"ABCDEFGHIJ@developer.gserviceaccount.com\", \"client_id\": \"ABCDEFGHIJ.apps.googleusercontent.com\", \"type\": \"service_account\"}"}'

Options

format "csv" or "tsv"
compression "" or "gz"
header_line true or false
delimiter "," or "\t" or "|"
null_string "" or "\N"
newline "CRLF" or "CR" or "LF"
json_keyfile need to escape newline '\n" with backslash

Last modified: Feb 24 2017 09:41:25 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.