SFTP V2 Server Export Integration

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

This integration enables uploading data to an SFTP server with support for

• multiple file formats
• data compression
• server hand-shaking algorithms

Prerequisites and Limitations

Prerequisites

• Basic knowledge of Treasure Data, including the toolbelt
• An SFTP server

Limitation

• Path prefix cannot contain any of the following characters: * ?

Static IP Address of Treasure Data Integration

If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.

Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/

Basic Usage

The following figure illustrates a typical usage scenario that involves exporting data from Treasure Data Agents into an SFTP or similar server.

Create a New Connection

You must create and configure the data connection on TD Console before running your query. As part of the data connection, you provide authentication to access the integration using the following steps:

1. Open the TD Console.
2. Navigate to Integrations Hub > Catalog.
3. Search for SFTP_V2 and then select Create Authentication.
4. Provide the connection credentials.

Authentication Fields

1. Select Continue, enter a name for the Authentication.
2. Select Done.

Configure a Query Result for Export

The TD Console supports multiple ways to export data. Complete the following steps to export data from the Data Workbench.

1. Navigate to Data Workbench > Queries.
2. Select New Query and define your query.
3. Select Export Results and provide the data export configuration.
4. Use an existing integration or create a new one.
5. Configure the export parameters and select Done.

Export Configuration Parameters

Activate a Segment in Audience Studio

You can also send segment data to the target platform by creating an activation in the Audience Studio.

1. Navigate to Audience Studio.
2. Select a parent segment.
3. Open the target segment, right-mouse click, and then select Create Activation.
4. In the Details panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters.
5. Customize the activation output in the Output Mapping panel.

• Attribute Columns
  • Select Export All Columns to export all columns without making any changes.
  • Select + Add Columns to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select + Add Columnsto add new columns for your activation output.
• String Builder
  • + Add string to create strings for export. Select from the following values:
    • String: Choose any value; use text to create a custom value.
    • Timestamp: The date and time of the export.
    • Segment Id: The segment ID number.
    • Segment Name: The segment name.
    • Audience Id: The parent segment number.

6. Set a Schedule.

• Select the values to define your schedule and optionally include email notifications.

7. Select Create.

(Optional) Export Integration Using the CLI

The TD Toolbelt can trigger exporting the Query Result to SFTP from the CLI. You need to specify the parameters for the export job using the --result option of the td query command. For more information, refer to Querying and Importing Data to Treasure Data from the Command Line.

The format of the option is JSON and the general structure is as follows:

• Authentication using user & password:

{
  "type": "sftp_v2",
  "host": "IP or host of SFTP server",
  "port": "The open port",
  "path_prefix": "/upload/2024Aug29/TC01.csv",
  "temp_file_threshold": "The size of tempfile",
  "timeout": 600,
  "user_directory_is_root": false,
  "rename_file_after_upload": true,
  "auth_method": "password",
  "username": "login user",
  "password": "password of user when auth_method is password"
}

• Authentication using public/privater key-pair

{
  "type": "sftp_v2",
  "host": "IP or host of SFTP server",
  "port": "The open port",
  "path_prefix": "/upload/2024Aug29/TC01.csv",
  "temp_file_threshold": "The size of tempfile",
  "timeout": 600,
  "user_directory_is_root": false,
  "rename_file_after_upload": true,
  "auth_method": "key_pair",
  "secret_key_file": "Content of private key file",
  "secret_key_passphrase": "td@123"
}

CLI Parameters

td ... --result '{"type":"sftp_v2", "secret_key_file": "-----BEGIN OPENSSH PRIVATE KEY-----\nline1\nline2\nline3\n-----END OPENSSH PRIVATE KEY-----\n\n"}'

-----BEGIN OPENSSH PRIVATE KEY-----
line1
line2
line3
-----END OPENSSH PRIVATE KEY-----

-----BEGIN OPENSSH PRIVATE KEY-----line1line2line3-----END OPENSSH PRIVATE KEY-----

quote_policy: ALL,
quote: "'",

'id'|'column01'|'column02'|'column03'|'column04'|'column05'
'1'|'data01'|'data02'|'data03'|'data04'|'data05'

quote_policy: MINIMAL,
quote: "'",

id|column01|column02|column03|column04|column05
1|data01|data02|data03|data04|data05

quote_policy: ALL,
quote: "'",

'id'|'column01'|'column02'
'1'|'data01'|'data02'

quote_policy: ALL,
quote: "\"",

"id"|"column01"|"column02"
"1"|"data01"|"data02"

Example Usage

1. Using password authentication

   td --database your_db --wait "SELECT email FROM (VALUES ('test01@test.com')) tbl(email)" \
   --type presto \
   --result '{"type":"sftp_v2","host":"your_ip","port": 22,"auth_method":"password","username":"user_name","password":"your_pass", "user_directory_is_root": true, "rename_file_after_upload": true,"path_prefix":"/sftp/2024aug/test.csv"}'

2. Using password authentication and data compression

   td --database your_db --wait "SELECT email FROM (VALUES ('test01@test.com')) tbl(email)" \
   --type presto \
   --result '{"type":"sftp_v2","host":"your_ip","port": 22,"auth_method":"password","username":"user_name","password":"password","user_directory_is_root":true,"path_prefix":"/sftp/2024aug/test.csv","rename_file_after_upload":true,"format": "csv", "compression": "gz","header_line":true,"quote_policy":"ALL","delimiter":"\t","null_string":"","newline":"CRLF","temp_file_threshold":0,"sequence_format":".%01d%01d"}'

3. Using Public/Private key authentication with password phrase

   Private key file content (password phrase is 123):

   -----BEGIN OPENSSH PRIVATE KEY-----
   line1
   line2
   line3
   -----END OPENSSH PRIVATE KEY-----

   Format for CLI:

   -----BEGIN OPENSSH PRIVATE KEY-----\nline1\nline2\nline3\n-----END OPENSSH PRIVATE KEY-----\n\n

   The CLI command:

   td --database your_db --wait "SELECT email FROM (VALUES ('test01@test.com')) tbl(email)" \
   --type presto \
   --result '{"type":"sftp_v2","host":"your_ip","port": 22,"auth_method":"key_pair","username":"user_name", "path_prefix": "/sftp/2024aug/test.csv","sequence_format":"","max_connection_retry":5,"secret_key_passphrase":"123","timeout":600,"secret_key_file":"-----BEGIN OPENSSH PRIVATE KEY-----\nline1\nline2\nline3\n-----END OPENSSH PRIVATE KEY-----\n\n"}'

4. Create the new customer

   td --database luan_db --wait "SELECT customer_list_id,first_name,last_name,birthday,company_name,email,fax,gender,job_title,phone_business,phone_home,phone_mobile,second_name,credentials FROM (VALUES ('site_1','fname_201','lname_1','1990-05-12','Test Company',

5. Using the encryption encoder

   td query -d luan_db -w "select * from luan_db.sfcc_500k" -T presto --result '{
     "type": "sftp_v2",
     "host": "your host server",
     "auth_method": "password",
     "username": "your user name",
     "password": "your password",
     "path_prefix": "cli/test_27_3_500K.csv.bz2.gpg",
     "file_ext": ".gpg",
     "rename_file_after_upload": false,
     "formatter": {
       "type": "csv",
       "quote_policy": "MINIMAL",
       "delimiter": ",",
       "null_string": "null",
       "newline": "\n"
     },
     "compression": "encrypt_pgp",
     "public_key": "your public key",
     "key_identifier": "your key identifier",
     "armor": true/false,
     "compression_type": "bzip2/gzip/none"
   }'

(Optional) Other considerations

Within Treasure Workflow, you can specify the use of this integration to export data. For sample workflows, view Treasure Boxes.

• The Result Export can be scheduled to upload data to a target destination periodically
• Most import and export integrations can be added to a TD Workflow to be involved in a more advanced data pipeline. Refer here for more detailed information.

References

• The Embulk-encoder-Encryption document: https://docs.treasuredata.com/articles/#!project-integrations/~embulk-encoder-encryption-pgp

FAQ for the SFTP Data Connector

Q: I cannot connect to my SFTP server. What can I do?

A: Here are some things to check:

• Verify that your protocol is valid. If you intended to use SFTP, you can use this integration. If you intended to use FTP/FTPS, try to connect with the FTP Server Import Integration.
• If you are using a firewall, check your accepted IP range and port. Server administrators sometimes change the default port number from TCP 22 for security reasons.
• Be sure that your private key has an OpenSSH format. Treasure Data does not support other formats like “PuTTY”.
• Treasure Data does not support the default format of the private key since OpenSSH 7.8. Re-generate the key using '-m PEM' option.

Q: I cannot find my uploaded file on the SFTP server. What happened?

A: If your SFTP server does not support Append mode, the integration may not be able to update the file with the error "SFTP Exception: no such file."  For stability reasons, the integration uploads the file content in batches. Consequently, you will need to enable the Append mode on your SFTP server. If Append mode cannot be enabled on your SFTP server, you will need to set temp_file_threshold to 0 to bypass this feature.

Note: Please ensure that you compress your file before encrypting and uploading.

1. When you decrypt using non-built-in encryption, the file will return to a compressed format such as .gz or .bz2.

2. When you decrypt using built-in encryption, the file will return to raw data.