# Gigya Import Integration

You can ingest data of [Accounts](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/b32ce0918af44c3ebd7e96650fa6cc1d.md), [Profile Management](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/81104a226fa04009870a898a77f58ca7.md), [Data Store](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/415206b370b21014bbc5a10ce4041860.md), and [Audit Log](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/4143815a70b21014bbc5a10ce4041860.md) from Gigya (SAP Customer Data Cloud) into Treasure Data.

## Prerequisites

- Basic knowledge of Treasure Data, including the [TD toolbelt](https://toolbelt.treasuredata.com/)
- A Gigya account
- Basic knowledge about [Query Syntax Specification](https://developers.gigya.com/display/GD/accounts.search+REST#accounts.searchREST-QuerySyntaxSpecification) in order to obtain data from Accounts objects
- Review [About Incremental Loading](/int/about-incremental-loading)


Incremental loading uses the maximum value (max value) in the specified incremental column to load all records until the max value for the first execution and subsequent runs imports records from (max value +1) from the previous run to the current time when the job runs, which becomes the new max value.

## Supported

- Select clause
- From clause
- Where clause
- Group By clause


## Limitations and Requirements

- **START**, **CONTAINS** and **WITH** keyword is not supported
- **COUNTERS** is not supported
- **LIMIT** clause in the query will be removed automatically(**SELECT * FROM ACCOUNTS LIMIT 100)**returns all records ignoring the limit
- Query with aggregate functions (***sum, min, max, avg, sum_of_squares, variance, std***) and **Group By** clause only could ingest the first page
- If you enter an invalid object in **FROM**clause (i.e ***SELECT * FROM unexisted***) the query will be automatically fall back to the accounts object
- Columns names are case-sensitive and Objects name are case-insensitive


### Query Syntax Limitation

Treasure Data supports the following SQL query syntax for Gigya:

- Select clause
- From clause
- Where clause
- Group By clause
- START, CONTAINS, and WITH keywords are not supported.
- COUNTERS is not supported.
- a LIMIT clause in the query is removed automatically.
SELECT * FROM ACCOUNTS LIMIT 100 returns all records and ignores the limit.
- A query with aggregate functions (sum, min, max, avg, sum_of_squares, variance, std ) and Group By clause can only ingest the first page.
- Columns names are case-sensitive.
- Object names are case-insensitive.
- The Incremental Column should be numeric or timestamp.


### For Account Data Source Limitation

- Your data query is limited to two objects: accounts, and email accounts.
- If you enter an invalid object in a FROM clause, the query automatically falls back to the accounts object.
For example, `SELECT * FROM does_not_exist` , substitutes accounts for `does_not_exist`.
- Supported incremental columns for accounts objects are:[lastLogin, registered, oldestDataUpdatedTimestamp, lastUpdated, verifiedTimestamp, oldestDataUpdated, lastUpdatedTimestamp, created, createdTimestamp, verified, registeredTimestamp, lastLoginTimestamp, lockedUntil]
- Supported incremental columns for emailAccounts are: [lastUpdated, lastUpdatedTimestamp, created, createdTimestamp]
- Referenced link: [accounts.search REST](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/b32ce0918af44c3ebd7e96650fa6cc1d.md)


### For Profile Management Data Source Limitation

- Your data query is limited to two objects, accounts, and email accounts.
- If you enter an invalid object in a FROM clause, the query automatically falls back to the accounts object.
For example, `SELECT * FROM does_not_exist` , substitutes accounts for `does_not_exist`.
- Supported incremental columns for accounts objects are:[lastLogin, registered, oldestDataUpdatedTimestamp, lastUpdated, verifiedTimestamp, oldestDataUpdated, lastUpdatedTimestamp, created, createdTimestamp, verified, registeredTimestamp, lastLoginTimestamp, lockedUntil]
- Supported incremental columns for emailAccounts are:[lastUpdatedTimestamp, created, createdTimestamp, lastUpdated]
- Referenced link: [ids.search REST](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/81104a226fa04009870a898a77f58ca7.md)


### For Data Store Data Source Limitations

- If you enter an invalid object in a FROM clause, you will receive error [400006] Invalid parameter value: Invalid argument: accounts type not allowed
- Depending on the schema of the target Data Store then incremental columns will be varied. But you could enter an invalid column name and see the error return from TD Console to know what is an acceptable column name.
- Referenced link: [ds.search REST](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/415206b370b21014bbc5a10ce4041860.md)


### For Audit Log Data Source Limitations

- Supported incremental columns are: [@timestamp]
- Referenced link: [audit.search](https://help.sap.com/viewer/8b8d6fffe113457094a17701f63e3d6a/GIGYA/en-US/4143815a70b21014bbc5a10ce4041860.md)


## Incremental Loading and Numeric and Timestamp Columns

Incremental loading uses the maximum value (max value) in the specified incremental column to load all records till max value for the first execution and subsequent runs import records from (max value +1) from the previous run to the current time when the job runs (which becomes the new max value).

Support for:

- Incremental columns of numeric or timestamp type
- Incremental columns for Accounts objects are:[lastUpdated, lastUpdatedTimestamp, created, createdTimestamp]
- Incremental columns for EmailAccounts are:
[lastLogin, registered, oldestDataUpdatedTimestamp, lastUpdated, verifiedTimestamp, oldestDataUpdated, lastUpdatedTimestamp, created, createdTimestamp, verified, registeredTimestamp, lastLoginTimestamp, lockedUntil]
- Supported incremental columns for auditLog are: [@timestamp]


## Use the TD Console to Create Your Connection

### Obtain your API Key, User Key, and Secret Key From Gigya

1. Follow the instruction in [Creating and Managing Applications](https://developers.gigya.com/display/GD/Signing+Requests+to+SAP+Customer+Data+Cloud#SigningRequeststoSAPCustomerDataCloud-CreatingandManagingApplications) to create your own application and obtain the **App User Key** and **Secret Key**.
2. Follow the instructions in [API Key and Site Setup](https://developers.gigya.com/display/GD/APIs+and+SDKs#APIsandSDKs-APIKeyandSiteSetup) to obtain your **API Key**.
3. Follow the instructions to determine your [Data Center](https://developers.gigya.com/display/GD/Finding+Your+Data+Center#FindingYourDataCenter-DataCenters).


### Create a New Connection

When you configure a data connection, you provide authentication to access the integration. In Treasure Data, you configure the authentication and then specify the source information.

1. Open TD Console.
2. Navigate to Integrations Hub ->  Catalog
3. Search and select Gigya (SAP Customer Data Cloud).


![](/assets/image2021-5-25_14-44-41.98720c1031155d78df3c5e939cf2cfbe0d4b74e1a0ebbc199572f82a93a8955e.236f7714.png)

The following dialog opens.

![](/assets/image2021-4-27_8-59-59.c1f19586bd853c61e891beb3085827a3ea92c634949a9cef4796e99db38d7474.236f7714.png)

1. Choose your account Data Center.
2. Type values for the following:


- API Key
- User Key
- User Secret


![](/assets/image2021-4-27_9-13-2.c9aa87cc32a42f315c85b097ccf634eff14662d56b3dfba341b6a90cf0b21d10.236f7714.png)

1. Select **Continue**.
2. Enter a name for your connection and select **Done.**


![](/assets/image2021-4-27_9-19-20.0251626fe2dbb06d6cd6a82282393d7bfd525d12e1a8d651ca3c983ff1025585.236f7714.png)

### Transfer Your Gigya Accounts Data to Treasure Data

After creating the authenticated connection, you are automatically taken to the Authentications tab.

1. Search for the connection you created and select **New Source**.


![](/assets/image2021-4-27_9-22-44.ff977a0a1de91a5bd4ac0859362faa6aa6a63e1d68b6c9d91be23e8967cbceec.236f7714.png)

1. Name the **Source.**
2. Select **Next**.
3. In the Source Table, edit the parameters.


![](/assets/image2021-12-16_9-48-7.6b4c32d58610b29a3d8bcbadab308a723f5d8e70c334249e82db71a9ca9af539.236f7714.png)

| **Parameters** | **Description** |
|  --- | --- |
| **Data Source** | Target data source. Current support: Accounts, Profile Management, Data Store and Audit Log |
| **Query** | Gigya's query to ingest data. Depend on your target object, the query would be variant.  For Account and Profile Management data source, only support on accounts and emailAccounts object (sample query: `Select * From accounts`)  For Data Store it would be whatever your object is (sample query: `Select * From my_data`)  For Audit Log, only support for auditLog object (sample query: `Select * From auditLog`) |
| **Fields To Exclude** | **Due to Gigya's API specification**, it is not possible to specify the columns to be included in the SELECT statement. This parameter can be used to remove unnecessary columns. |
| **Batch Size** | The maximum number of records to fetch in a single API call. The maximum value is 10000 and the minimum value is 10. When you customize the batch size, consider the following: a smaller value will let the API return faster but will cause more API calls. |
| **Incremental** | When running on a schedule, the next import ingests only the data that was updated after the last run based on the value of the Incremental Column. |
| **Incremental Column** | Which data object's column on which to perform the incremental transfer.  For Account and Profile Management data source, suggested values are: created, createdTimestamp, updated, and updatedTimestamp.  For Data Store data source, suggested values are: numeric or datetime colum  For Audit Log data source, suggested values are: @timestamp |


1. In **Data Settings** dialog, you can edit data settings or opt to skip this step.


![](/assets/screenshot-2025-01-06-at-11.22.05.f73041e49fc1f9443fc53aa7c7f46b62da9741e7d71e1caaaeb466f0955200a9.236f7714.png)

By default, the connector can detect data type automatically. However, the user can customize data type for import attribute fields via *column_options* parameter. Supported data types mapping:

- boolean
- long
- double
- string
- timestamp
- json


### Data Preview

You can see a [preview](/products/customer-data-platform/integration-hub/batch/import/previewing-your-source-data) of your data before running the import by selecting Generate Preview. Data preview is optional and you can safely skip to the next page of the dialog if you choose to.

1. Select **Next**. The Data Preview page opens.
2. If you want to preview your data, select **Generate Preview**.
3. Verify the data.


### Data Placement

For data placement, select the target database and table where you want your data placed and indicate how often the import should run.

1. Select **Next.** Under Storage, you will create a new or select an existing database and create a new or select an existing table for where you want to place the imported data.
2. Select a **Database** > **Select an existing** or **Create New Database**.
3. Optionally, type a database name.
4. Select a **Table**> **Select an existing** or **Create New Table**.
5. Optionally, type a table name.
6. Choose the method for importing the data.
  - **Append** (default)-Data import results are appended to the table.
If the table does not exist, it will be created.
  - **Always Replace**-Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created.
  - **Replace on New Data**-Only replace the entire content of an existing table with the result output when there is new data.
7. Select the **Timestamp-based Partition Key** column.
If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter.
8. Select the **Timezone** for your data storage.
9. Under **Schedule**, you can choose when and how often you want to run this query.


#### Run once

1. Select **Off**.
2. Select **Scheduling Timezone**.
3. Select **Create & Run Now**.


#### Repeat Regularly

1. Select **On**.
2. Select the **Schedule**. The UI provides these four options: *@hourly*, *@daily* and *@monthly* or custom *cron*.
3. You can also select **Delay Transfer** and add a delay of execution time.
4. Select **Scheduling Timezone**.
5. Select **Create & Run Now**.


After your transfer has run, you can see the results of your transfer in **Data Workbench** > **Databases.**