This article explains how to send job results directly to your Dropbox account.

Prerequisites

  • Basic knowledge of Treasure Data, including the toolbelt.

  • A Dropbox account 

  • Authorized Treasure Data account access

Use the TD Console to Create Your Connection

Create a New Connection

When you configure a data connection, you provide authentication to access the integration. In Treasure Data, you configure the authentication and then specify the source information.

  1. Navigate to Integrations Hub >  Catalog. Search and select Dropbox.

  2. The following dialog opens. Access to Treasure Data Dropbox requires OAuth2 authentication. The authentication requires that users manually connect their Treasure Data account to their respective Dropbox account.

    To authenticate, select Click here to connect to a new account.


  3. Log into your Dropbox account in the popup window and grant access to the Treasure Data app (You only need to do this the first time.)


  4. You will be redirected back to TD Console. Repeat the first step (Create a new connection) and choose your new OAuth connection.


  5. Name your new Dropbox Connection. Select Done.


Configure Output Results to the Data Connection

In this step, you create or reuse a query. In the query, you configure the data connection. 

  1. Navigate to Data Workbench > Queries.

  2. Select the query that you plan to use to export data OR create a new query, if needed.

  3. Select Export Results to specify the results export target.

Specify the Result Export Target

  1. After selecting Export Results, the following dialog opens.

  2. Type the connection name in the search box to filter and select your Dropbox connection.


  3. Edit all the information.


     

Parameter

Description

Folder Path

directory of the folder in Dropbox

File Name

destination file name without extension

Format

file extension

Compression

Indicate whether the result is to be compressed or not. Available options: None, gz, and bzip2.

Header line

Select if the exported data has the column name as the header line.

Null String

Use this value to represent NULL values. Available options:

  • Default: select if an empty string ( '' ) is used to represent the Null value for file format CSV, and \N is used for the TSV format.

  • Empty string

  • \N NULL

  • null 

End-of-line character

the character at the end of lines. Available options are CRLF, LF, and CR

Quote Policy

available options:

  • ALL: select if all values are enclosed by double quotes ("").

  • MINIMAL: select if any value that contains an embedded quote (") is presented with a consecutive pair of quotes (""). The MINIMAL quote policy is applied to CSV.

  • NONE: select if no escape for embedded quote is applied. By default, NONE is applied to TSV file format.

Max Retries

when an error occurs due to a Dropbox server error or network error, the number of retries attempted before aborting the upload.


Execute the Query

  1. Select Save to save the query with a name and run OR select Run to run the query.


  2. After the query has run successfully, the query result is automatically imported into the specified container destination.

Example of a Query

SELECT email, first_name, last_name, region, age, gender, website FROM (
 VALUES ('kate1@gmail.com', 'Kate', 'Tiny', 'Asia', '41 and above', 'female', 'google.com'),
 ('ronan@gmail.com', 'R', 'P', 'Americas', '21-30', 'male', 'google.com'),
 ('michelle@gmail.com', 'M', 'C', 'EMEA', '31-40', 'male', 'facebook.com')
) tbl (email, first_name, last_name, region, age, gender, website);


Optional: Use of Scheduled Jobs for Output

You can use Scheduled Jobs with Result Output, to periodically write the output result to a target destination that you specify.

Optional: Configure Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to output data.

timezone: UTC

_export:
  td:
    database: sample_datasets

+td-result-into-one-drive:
  td>: queries/sample.sql
  result_connection: your_connection_name
  result_settings:
    access_token: ###
    file_name: file01
    folder_path: /abc
    replace_existing: false
    format: csv
    compression: gz
    header_line: true
    null_string: default
    newline: CRLF
    quote_policy: null
    retry_count: 5
    retry_initial_wait_millis: 1000
    max_retry_wait_millis: 30000

 Select here for more information on using data connectors in the workflow to export data.

Use the CLI to create your connection

Install ‘td’ command

Install the Treasure Data Toolbelt.

For On-demand Jobs

Add the Dropbox result output destination by using the r/ -result option for the td query command:

td query -d sample_datasets -w 'SELECT host, path FROM www_access' --type presto -r '{"type":"dropbox", "access_token":"your_token","folder_path":"/abc","file_name":"test_file","replace_existing":null,"format":"csv","compression":"gz","header_line":true,"null_string":"default","newline":"CRLF","quote_policy":null,"retry_count":"5","retry_initial_wait_millis":"1000","max_retry_wait_millis":"300000"}'

For Scheduled Jobs

Add the Dropbox result output destination by using the r/ -result option for the td sched:create command:

td sched:create every_6_mins "*/6 * * * *" -d test_db -w 'SELECT id, via FROM table1' --type presto -r '{"type":"dropbox", "access_token":"your_token","folder_path":"/abc","file_name":"test_file","replace_existing":null,"format":"csv","compression":"gz","header_line":true,"null_string":"default","newline":"CRLF","quote_policy":null,"retry_count":"5","retry_initial_wait_millis":"1000","max_retry_wait_millis":"300000"}'

Appendix

FAQ for Export into Dropbox

Q: Dropbox file size limits?

  • All files uploaded to your Dropbox must be smaller than your storage space. For example, if your account has a storage quota of 2 GB, you can upload one 2 GB file or many files that add up to 2 GB. If you are over your storage quota, Dropbox will stop syncing.

  • Files uploaded through must be 350GB or smaller.

  • No labels