Learn more about FTP Server Import Integration.

This article explains how to send job results directly to your FTP server, with FTPS (explicit) or FTPES (implicit).

For sample workflows on how to export job results to your FTP server, view Treasure Boxes.

Prerequisites

  • Basic knowledge of Treasure Data, including the TD toolbelt.

  • An FTP server

Basic Usage


You can limit access to your endpoint by using a list of static IPs. Contact support@treasuredata.com if you need it.


Get your Credential for your FTP(S) Server

Obtain your credential to access your FTP(S) server.

Write the Query

Navigate to the TD Console query editor page and compile your query.

Specify the Result Export target

In the same window, select Add in the Result Export section and select FTP from the drop-down menu. Edit all the information, including your Host, Port, Username, Password.

If you want to connect to FTP Server with FTPS(explicit) or FTPES(implicit), select Use SSL? and select proper options.

Execute the Query

Either save the query with a name and run or just run the query. Upon successful completion of the query, its result will be automatically imported to the specified Container destination:


Usage from CLI

You can also use CLI for Result Export to FTP(S).

Example,


CLI for FTP

$ td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":21,"username":"xxxx","password":"xxxxx","path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto


CLI for FTPS (Implicit mode)

$ td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":990,"username":"xxxx","password":"xxxxx","passive_mode":true,"ascii_mode":true,"ssl":true,"ssl_explicit":false,"ssl_verify":false,"ssl_verify_hostname":false,"path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto


CLI for FTPES (Explicit mode)

$ td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":21,"username":"xxxx","password":"xxxxx","passive_mode":true,"ascii_mode":true,"ssl":true,"ssl_explicit":true,"ssl_verify":false,"ssl_verify_hostname":false,"path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto


Legacy mode(Deprecated)

We kept this feature for backward compatibility and will stop providing it in the near future.

For On-Demand Jobs

For on-demand jobs, just add the --result option to the td query command. After the job is finished, the results are sent to the specified FTP server.

$ td query \
  --result 'ftp://user:password@domain.com/path.csv' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

If a port number is not specified, the default value is 21.

The request body is in CSV format (RFC 4180) where the first line is a header with the column names, the new line is CRLF, the text encoding is UTF-8, and the quotation character is double-quoted.

_c0,_c1
a,b
c," d "
e,f

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. After every job run, the results are sent in the same manner as on-demand jobs.

$ td result:create myftp ftp://user:password@domain.com/
$ td sched:create hourly_count_example "0 * * * *" \
  -d testdb \
  --result myftp:path.csv \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"
  • No labels