Skip to content
/
Gigya Import Integration...
Last updated

Gigya Import Integration CLI

You can use the CLI to configure your connection.

Install the Treasure Data Toolbelt

Open a terminal and run the following command to install the newest Treasure Data Toolbelt.

Prepare load.yml File

Prepare load.yml. The in: section is where you specify what comes into the connector from Gigya and the out: section is where you specify what the connector puts out to the database in Treasure Data.

Provide your Gigya account access information as follows:

in:
  type: gigya
  data_center: US1
  authentication_mode: key_secret
  application_key: your_application_user_key
  secret_key: your_application_secret_key
  api_key: your_api_key
  data_source: account
  query: SELECT * FROM accounts
  fields_to_exclude: "XffFirstIp, httpReq"
  batch_size: 1000

Configuration keys and descriptions are as follows:

Config keyTypeRequiredDescription
typestringyesconnector type
data_centerstringyesspecifies your data center location (available values are US1, EU1, AU1, RU1, CN1)
authentication_modestringnoMethod for authentication, current support only key_secret
application_keystringyesYour application's user key
secret_keystringyesYour application's secret key
api_keystringyesYour API key
data_sourcestringnoYour target data source (available values are account, profile, data_store, audit)
querystringyesYour custom Gigya query
fields_to_excludestringnoDue to Gigya's API specification, it is not possible to specify the columns to be included in the SELECT statement. This parameter can be used to remove unnecessary columns.
batch_sizenumbernoMaximum number of records in a single batch

Preview the Data to be Imported

You can preview data to be imported using the command td connector:preview.

$ td connector:preview load.yml

Execute the Load Job

Use td connector:issue to execute the job. Processing might take a couple of hours depending on the data size. The following are required:

  • name of the schedule
  • cron-style schedule
  • database and table where their data will be stored
  • the Data Connector configuration file
td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column created_at
daily_xxxx_import

The preceding command assumes you have already created database(td_sample_db) and table(td_sample_table). If the database or the table do not exist in TD, this command will not succeed. You must create the database and table manually or use --auto-create-table option with td connector:issue command to auto-create the database and table:

td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column created_at --auto-create-table

It is recommended to specify --time-column option, because Treasure Data’s storage is partitioned by time. If the option is not given, the data connector selects the first long or timestamp column as the partitioning time. The type of the column, specified by --time-column, must be either of long or timestamp type. Use Preview results to check for the available column name and type. Generally, most data types have a last_modified_date column.

A time column is available at the end of the output.

td connector:issue load.yml --database td_sample_db --table td_sample_table
--time-column created_at

If your data doesn’t have a time column you can add it using the add_time filter. You add the "time" column by adding the add_time filter to your configuration file as follows.

in:
  type: xxxxx
  ...
filters:
- type: add_time
  from_value:
    mode: upload_time
  to_column:
    name: time
out:
  type: td

Find more information at add_time filter plugin.

If you have a field called time, you do not have to specify --time-column option.

$ td connector:issue load.yml --database td_sample_db --table td_sample

Scheduled Execution

You can schedule periodic data connector execution for periodic Gigya import. We configure our scheduler carefully to ensure high availability. By using this feature, you no longer need a cron daemon on your local data center.

For the scheduled import, the data connector for Gigya imports all objects that match the specified target.

Scheduled execution supports additional configuration parameters that control the behavior of the data connector during its periodic attempts to fetch data from Gigya:

  • incremental This configuration is used to control the load mode, which governs how the data connector fetches data from Gigya based on one of the native timestamp or numeric field associated with each object
  • incremental_columnn This configuration is used to define a based column to import into Treasure Data. You can define only one column for this field. Suggested values are created, createdTimestamp, updated, updatedTimestamp

Here’s an example of a load file using incremental mode

in:

  type: gigya

  data_center: US1

  authentication_mode: key_secret

  application_key: your_application_user_key

  secret_key: your_application_secret_key

  api_key: your_api_key

  data_source: account

  batch_size: 1000

  query: SELECT * FROM accounts

  incremental: true

  incremental_column: created

filters:

- type: add_time

  from_value:

    mode: upload_time

  to_column:

    name: time

Create the schedule

A new schedule can be created using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the data connector configuration file are required.

The cron parameter accepts these options: @hourly, @daily and @monthly.

By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The --timezone option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are not supported and may lead to unexpected schedules.

You can create scheduled job to import using the command td connector:create to run daily.

td connector:create connector_name @daily connector_database connector_table load.yml

Modes for the plugin

You can specify import mode in the out section of the load.yml file.

The out: section controls how data is imported into a Treasure Data table. For example, you may choose to append data or replace data in an existing table in Treasure Data.

Output modes are ways to modify the data as the data is placed in Treasure Data.

  • Append(default): Records are appended to the target table.
  • Replace (available In td 0.11.10 and later): Replaces data in the target table. Any manual schema changes made to the target table remain intact.

Examples:

in:

  ...

out:

  mode: append
in:

  ...

out:

  mode: replace
Previous page
Next page