Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
maxLevel32


Prerequisites

  • Basic knowledge of Treasure Data.

...

Numbered Headings
start-numbering-with1
start-numbering-ath5
Open TD Console.
Navigate to Integrations Hub  Catalog.
Search for and select SFTP_V2.

Select Create Authentication.
The following dialog opens. Edit the parameters. Select Continue.

     

ParametersDescription
HostThe host information of the remote SFTP instance, for example an IP address.
PortThe connection port on the remote SFTP instance, the default is 22.
UserThe user name used to connect to the remote SFTP instance.
Authentication modeThe way you choose to authenticate with your SFTP server.
Secret key fileRequired if 'public / private key pair' is selected from `Authentication Mode`. (ecdsa key type is supported.)
Passphrase for secret key file(Optional) If required, provide a passphrase for the provided secret file.
Retry limitNumber of times to retry a failed connection (default 10).
Timeout

Connection timeout in seconds (default 600).


Enter a name for your connection.
Choose to share the authentication with others or not. 
Select Continue.


...

Code Block
linenumberstrue
in:
  type: sftp_v2
  host: <HOST>
  port: <PORT, default is 22>
  auth_method: key_pair
  user: <USER>
  secret_key_file:
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      Proc-Type: 4,ENCRYPTED
      DEK-Info: AES-128-CBC...
      ...
      -----END RSA PRIVATE KEY-----
  secret_key_passphrase: <PASSPHRASE>
  user_directory_is_root: true
  timeout: 600
  path_prefix: /path/to/sample
out:
  mode: append
  exec: {}


`secret_key_file` requires OpenSSH format.

Password Authentication

Create seed.yml with the following content.

Code Block
linenumberstrue
in:
  type: sftp_v2
  host: <HOST>
  port: <PORT, default is 22>
  auth_method: password
  user: <USER>
  password: <PASSWORD>
  user_directory_is_root: true
  timeout: 600
  path_prefix: /path/to/sample
out:
  mode: append
  exec: {}


You can use the following special characters in the password: "#$!*@"


The SFTP_v2 integration imports all files that match the specified prefix. path_prefix must point to file or folder (e.g. path_prefix: path/to/sample–> path/to/sample/201501.csv.gz, path/to/sample/201502.csv.gz, …, path/to/sample/201505.csv.gz).

...

Code Block
linenumberstrue
$ td connector:create \
    daily_import \
    "10 0 * * *" \
    td_sample_db \
    td_sample_table \
    load.yml \
    --time-column created_at


The `cron` parameter also accepts three special options: `@hourly`, `@daily` and `@monthly`.

By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules.

...