FTP Server Export Integration

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

Learn more about FTP Server Import Integration.

This article explains how to send job results directly to your FTP server, with FTPS (explicit) or FTPES (implicit).

For sample workflows on how to export job results to your FTP server, view Treasure Boxes.

Prerequisites

• Basic knowledge of Treasure Data, including the TD toolbelt.
• An FTP server

Static IP Address of Treasure Data Integration

If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.

Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/

Get your Credential for your FTP(S) Server

Obtain your credential to access your FTP(S) server.

Write the Query

Navigate to the TD Console query editor page and compile your query.

Specify the Result Export target

In the same window, select Add in the Result Export section and select FTP from the drop-down menu. Edit all the information, including your Host, Port, Username, Password.

If you want to connect to FTP Server with FTPS(explicit) or FTPES(implicit), select Use SSL? and select proper options.

With PGP Encryption Configuration

(Optional) Schedule Query Export Jobs

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

Treasure Data's scheduler feature supports periodic query execution to achieve high availability.

When two specifications provide conflicting schedule specifications, the specification requesting to execute more often is followed while the other schedule specification is ignored.

For example, if the cron schedule is '0 0 1 * 1', then the 'day of month' specification and 'day of week' are discordant because the former specification requires it to run every first day of each month at midnight (00:00), while the latter specification requires it to run every Monday at midnight (00:00). The latter specification is followed.

Scheduling your Job Using TD Console

1. Navigate to Data Workbench > Queries

2. Create a new query or select an existing query.

3. Next to Schedule, select None.

4. In the drop-down, select one of the following schedule options:

   Drop-down Value	Description
   Custom cron...	Review Custom cron... details.
   @daily (midnight)	Run once a day at midnight (00:00 am) in the specified time zone.
   @hourly (:00)	Run every hour at 00 minutes.
   None	No schedule.

Custom cron... Details

 *    *    *    *    *
 -    -    -    -    -
 |    |    |    |    |
 |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 |    |    |    +---------- month (1 - 12)
 |    |    +--------------- day of month (1 - 31)
 |    +-------------------- hour (0 - 23)
 +------------------------- min (0 - 59)

The following named entries can be used:

• Day of Week: sun, mon, tue, wed, thu, fri, sat.
• Month: jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec.

A single space is required between each field. The values for each field can be composed of:

1. (Optional) You can delay the start time of a query by enabling the Delay execution.

Execute the Query

Save the query with a name and run, or just run the query. Upon successful completion of the query, the query result is automatically exported to the specified destination.

(Optional) You can delay the start time of a query by enabling the Delay execution.

Activate a Segment in Audience Studio

You can also send segment data to the target platform by creating an activation in the Audience Studio.

1. Navigate to Audience Studio.
2. Select a parent segment.
3. Open the target segment, right-mouse click, and then select Create Activation.
4. In the Details panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters.
5. Customize the activation output in the Output Mapping panel.

• Attribute Columns
  • Select Export All Columns to export all columns without making any changes.
  • Select + Add Columns to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select + Add Columnsto add new columns for your activation output.
• String Builder
  • + Add string to create strings for export. Select from the following values:
    • String: Choose any value; use text to create a custom value.
    • Timestamp: The date and time of the export.
    • Segment Id: The segment ID number.
    • Segment Name: The segment name.
    • Audience Id: The parent segment number.

6. Set a Schedule.

• Select the values to define your schedule and optionally include email notifications.

7. Select Create.

Usage from CLI

You can also use CLI for Result Export to FTP(S).

Example,

CLI for FTP

td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":21,"username":"xxxx","password":"xxxxx","path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto

CLI for FTPS (Implicit mode)

td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":990,"username":"xxxx","password":"xxxxx","passive_mode":true,"ascii_mode":true,"ssl":true,"ssl_explicit":false,"ssl_verify":false,"ssl_verify_hostname":false,"path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto

CLI for FTPES (Explicit mode)

td query --result '{"type":"ftp","host":"xx.xx.xx.xx","port":21,"username":"xxxx","password":"xxxxx","passive_mode":true,"ascii_mode":true,"ssl":true,"ssl_explicit":true,"ssl_verify":false,"ssl_verify_hostname":false,"path_prefix":"/path/to/file","file_ext":".csv","sequence_format":"","header_line":true,"quote_policy":"MINIMAL","delimiter":",","null_string":"","newline":"CRLF"}' -d sample_datasets "select * from www_access" -T presto

Legacy mode(Deprecated)

For On-Demand Jobs

For on-demand jobs, just add the --result option to the td query command. After the job is finished, the results are sent to the specified FTP server.

td query \
  --result 'ftp://user:password@domain.com/path.csv' \
  -w -d testdb \
  "SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

If a port number is not specified, the default value is 21.

The request body is in CSV format (RFC 4180) where the first line is a header with the column names, the new line is CRLF, the text encoding is UTF-8, and the quotation character is double-quoted.

_c0,_c1
a,b
c," d "
e,f

For Scheduled Jobs

For scheduled jobs, just add the --result option when scheduling a job. After every job run, the results are sent in the same manner as on-demand jobs.

td result:create myftp ftp://user:password@domain.com/

td sched:create hourly_count_example "0 * * * *" \
-d testdb \
--result myftp:path.csv \
"SELECT code, COUNT(1) AS cnt FROM www_access GROUP BY code"

References

• The Embulk-encoder-Encryption document: https://docs.treasuredata.com/articles/#!project-integrations/~embulk-encoder-encryption-pgp

FAQ for the FTP Data Connector

Note: Please ensure that you compress your file before encrypting and uploading.

1.  When you decrypt using non-built-in encrypti on, the file will return to a compressed format such as .gz or .bz2.

2. When you decrypt using built-in encrypti on, the file will return to raw data.