# Shopify Import Custom Query Integration V2 The Shopify Import Connector V2—Custom query is designed to work with Shopify’s GraphQL Admin API and allow users to define their own desired queries. Continue to develop the features previously implemented in Shopify Import Integration V2. Add a new import operation: - Custom Query Import - Support 4 types of queries: - Query - Normal from object in plural - Query - Normal from object in singular (must have filter by a required field, such as id) - Query with NODES in plural - Query with NODE in single ## Prerequisites - Basic Knowledge of Treasure Data - Basic knowledge of Shopify ## Requirements and Limitations - Metafield Import Restrictions - Maximum 250 metafields per product - Maximum 250 metafields per product variant - Product Variants imports no longer supports incremental loading by the created_at timestamp. It now only supports loading by the updated_at timestamp. ## Static IP Address of Treasure Data Integration If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection. Please find the complete list of static IP addresses, organized by region, at the following link: [https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/](https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/) ## Import Shopify reports via TD Console ### Create Authentication Your first step is to create a new authentication with a set of credentials. 1. Select **Integrations Hub**. 2. Select **Catalog**. ![](/assets/integrationshub-catalog2.e33c0a4c7d81c40cc83dd056c2143b97b1406220e213cab14ef349d69412ffef.eedebb45.png) 3. Search for your Shopify in the Catalog; hover your mouse over the icon and select **Create Authentication**. ![](/assets/shopifyi1.4f504665b75eb5d2d1b5f353f6a747aa3f7c1b5d2e55c08b2da6278ecd120a7b.eedebb45.png) ![](/assets/shopifyi2.63ac6cca8a3716b1317b6e19b72e58c952551c47e5b05c6eac4063d4d346b6a2.eedebb45.png) 1. Ensure that the **Credentials** tab is selected and then enter credential information for the integration. ![](/assets/shopify2.39bc5212a8d0c245cbdc7397ee368a94fce6456486aeae4c1da24e7d2b886116.0efdb04c.png) **New Authentication Fields** | Parameter | Description | | --- | --- | | Store name | The store identifier for your Shopify store. This can be entered in two formats: - Full store URL: for example https://*mountbaker*.myshopify.com - Store name only: for example: *mountbaker*If you’re using just the store name, it should be the unique identifier of your store without the `.myshopify.com` domain. | | Admin API access token | The access token used to authenticate with Shopify’s Admin API. This token can be generated from your Shopify admin panel under **Apps > Develop apps > Create an app > Configure Admin API scopes**. The token needs the appropriate permissions to perform operations like managing metafields. | 1. Select **Continue**. 2. Enter a name for your authentication, and then select **Done**. ### Create Custom Query Source After the authentication becomes available on the console, configure your import job. 1. Open TD Console. 2. Navigate to **Integrations** **Hub > Authentications**. 3. Locate your Shopify authentication and select **New Source**. ### Create a Connection 1. Type a source name in the Data Transfer Name field. 2. Select **Next**. | Parameter | Description | | --- | --- | | Data Transfer Name | Enter a name for your transfer. | | Authentication | This field contains the name of the authentication that will be used to connect with Shopify. | ### Identify a Source Table 1. Configure the fields for the source table ![](/assets/screenshot-2025-05-06-at-13.27.54.44a6e83a7378638f2f5f2c2f736373c846ea1e278ef39ecfd801bb4608e3afd4.41035e72.png) | Field | Description | | --- | --- | | Source | A drop-down menu that contains the following Shopify objects: - Product - Product Variants - Meta Fields - Custom Query Select the Shopify object that contains the data you want to import from your Shopify store. | | Incremental? | When enabled, the connector will only import new or updated data since the last import run, making subsequent imports more efficient. | | Incremental field | Choose which timestamp to use for incremental loading:- created_at — Filter products by their creation date. - updated_at — Filter products by their last modification date. | | Custom data filter field | Choose which timestamp to use for the data filter. In the Custom Query source, we support using {{date_filter}} in the defined custom query to filter the data:- created_at — Filter products by their creation date. - updated_at — Filter products by their last modification date. | | Start date | The beginning timestamp from which to export data (Format: dd/mm/yyyy, hh:mm) | | End date | The ending timestamp where you want to finish exporting data (Format: dd/mm/yyyy, hh:mm). If left empty, the end date will be the current time. | | Custom Query | The designed user's own custom query. Can use 2 placeholder options:- ***{{incremental_filter}}**:*** Use in the custom query to use incremental (required). And input the start date and end date (optional). Example: ``` { products(first: 250, query: "{{incremental_filter}}") { pageInfo { hasNextPage endCursor } .... } } ``` - ***{{date_filter}}:*** Use in the custom query to use the date filter (optional). And input the start date and end date (optional). Example: ``` { products(first: 250, query: "{{date_filter}}") { pageInfo { hasNextPageendCursor } .... } } ``` - **Noted:** If you want to use pagination in a custom query. Please add in the custom query. ``` pageInfo {hasNextPageendCursor } ``` | 1. Select **Next**. ### Define Data Settings 1. Configure the data settings. ![](/assets/screenshot-2025-05-06-at-13.46.15.7052b5e417d51ffd1e1d8602a21c89a49c88d50b63521022d86774165e33483f.41035e72.png) | Parameter | Description | | --- | --- | | Retry Limit | The number of retries before the import fails. | | Initial retry time wait in millis | The initial time in milliseconds to wait before retrying. | | Max retry wait in millis | The maximum time in milliseconds to wait before retrying. | | Schema Settings | The schema was guessed from the sample data. Can modify the type and format before PREVIEW and RUN. | - **Noted:** - Only support takes the top-level 1 of the JSON field. Do not support parsing the key-value pairs of the JSON object. - Remember to change the JSON fields from string to JSON data type to get the data. Normally, the guessed field data type is a string. - Remember, DO NOT CHANGE the field name if not needed. - After you change anything in the custom query. You should check and edit the schema settings once again. ### Data Preview You can see a [preview](/products/customer-data-platform/integration-hub/batch/import/previewing-your-source-data) of your data before running the import by selecting Generate Preview. Data preview is optional and you can safely skip to the next page of the dialog if you choose to. 1. Select **Next**. The Data Preview page opens. 2. If you want to preview your data, select **Generate Preview**. 3. Verify the data. ### Data Placement For data placement, select the target database and table where you want your data placed and indicate how often the import should run. 1. Select **Next.** Under Storage, you will create a new or select an existing database and create a new or select an existing table for where you want to place the imported data. 2. Select a **Database** > **Select an existing** or **Create New Database**. 3. Optionally, type a database name. 4. Select a **Table**> **Select an existing** or **Create New Table**. 5. Optionally, type a table name. 6. Choose the method for importing the data. - **Append** (default)-Data import results are appended to the table. If the table does not exist, it will be created. - **Always Replace**-Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created. - **Replace on New Data**-Only replace the entire content of an existing table with the result output when there is new data. 7. Select the **Timestamp-based Partition Key** column. If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter. 8. Select the **Timezone** for your data storage. 9. Under **Schedule**, you can choose when and how often you want to run this query. #### Run once 1. Select **Off**. 2. Select **Scheduling Timezone**. 3. Select **Create & Run Now**. #### Repeat Regularly 1. Select **On**. 2. Select the **Schedule**. The UI provides these four options: *@hourly*, *@daily* and *@monthly* or custom *cron*. 3. You can also select **Delay Transfer** and add a delay of execution time. 4. Select **Scheduling Timezone**. 5. Select **Create & Run Now**. After your transfer has run, you can see the results of your transfer in **Data Workbench** > **Databases.** ## Import from Shopify reports via Workflow You can import data from Shopify reports via workflow using td_load>: src_id. If you have already created a source, you can run it; if you don't want to create a source, you can import it using a .yml file. ### Using a Source 1. Select **Integrations Hub>Sources**. 2. On the Filters pane on the far right of the screen, use the Integration Type drop-down menu to select **Shopify V2**. 3. In the Sources pane, identify the row that contains the source you want to use, and then use the more drop-down menu ( •••  icon) for that row to select **Copy Unique ID**. ![](/assets/26617502.09d9b84b0f1f752c7c95b0bc1c2d8e8b7302e5b91c6a3cb5f01309dadf53a604.25ec5a77.png) 1. Define a workflow task, and for td_load>: use the ID you copied in step 3. ```yaml +load: td_load>: unique_id_of_your_source database: ${td.dest_db} table: ${td.dest_table} ``` 1. Run the workflow. ### Using a .yml file 1. Identify your .yml file. If you need to create the .yml file, refer to the instructions found in [Create Seed Config File (seed.yml)](/int/amazon-s3-import-integration-v2#AmazonS3ImportIntegrationv2-CreateSeedConfigFile(seed.yml)). 2. Define a workflow task, and for td_load>: specify your .yml file. ```yaml +load: td_load>: config/daily_load.yml database: ${td.dest_db} table: ${td.dest_table} ``` 1. Run the workflow. #### Sample Workflow Code Visit [Treasure Boxes](https://github.com/treasure-data/treasure-boxes/tree/master/td_load/s3) for a sample workflow code. ## Import from Shopify via CLI (Toolbelt) Before setting up the integration, install the latest version of the [TD Toolbelt](https://toolbelt.treasuredata.com/). ### Prepare a Load File ```yaml in: type: shopify_v2 admin_api_access_token: xxxxxxxx target: custom_query store_name: xxxxxxx from_date: '2024-12-31T17:00:00.000Z' incremental_field: created_at incremental: true query: | { products(first: 10, query: "{{incremental_filter}}") { pageInfo { hasNextPage endCursor } edges { node { id title productType createdAt ... ... ... } } } } out: mode: replace ``` **Parameters Reference** | Name | Description | Value | Default Value | Required | | --- | --- | --- | --- | --- | | type | The source of the import. | "shopify_v2" | | Yes | | admin_api_access_token | The access token used to authenticate with Shopify’s Admin API. This token can be generated from your Shopify admin panel under **Apps > Develop apps > Create an app > Configure Admin API scopes**. The token needs the appropriate permissions to perform operations like managing metafields. | String | | Yes | | target | The source you want to collect the data from | String. "custom_query" | | Yes | | store_name | The store identifier for your Shopify store. This can be entered in two formats: - Full store URL: for example https://*mountbaker*.myshopify.com - Store name only: for example: *mountbaker* If you’re using just the store name, it should be the unique identifier of your store without the `.myshopify.com` domain. | String. | | Yes | | incremental | List of metrics your report should be grouped by. Allowable values depend on report type. | Boolean. | False | No | | incremental_field | Timestamp to use for incremental loading. | String. "created_at" or "updated_at" | created_at | No | | date_filter_field | Timestamp to use {{date_filter}} in the defined custom query to filter the data | String. "created_at" or "updated_at" | created_at | No | | start_date | The beginning timestamp from which to export data | String. Format: yyyy-MM-dd'T'HH:mm:ss.SS'Z' | | No | | end_date | The ending timestamp where you want to finish exporting data | String. Format: yyyy-MM-dd'T'HH:mm:ss.SS'Z' | | No | | query | The designed user's own custom query. Can use 2 placeholder options: - ***{{incremental_filter}}**:*** Use in the custom query to use incremental (required) - ***{{date_filter}}:*** Use in the custom query to use the date filter (optional). - **Noted:** If you want to use pagination in a custom query. Please add in the custom query. | String. | | Yes | To guess the data, use the `td connector:guess` command. (Should use guess first to guess the schema -> Modify schemas like expectation in "columns" property). ```bash $ td connector:guess guess.yml ``` ```yaml in: type: shopify_v2 admin_api_access_token: xxxxxxxx target: custom_query store_name: xxxxxxx from_date: '2024-12-31T17:00:00.000Z' incremental_field: created_at incremental: true query: | { products(first: 10, query: "{{incremental_filter}}") { pageInfo { hasNextPage endCursor } edges { node { id title productType createdAt ... ... ... } } } } out: mode: replace ``` To preview the data, use the `td connector:preview` command. ``` $ td connector:preview load.yml ``` ### Execute the Load Job It might take a couple of hours, depending on the size of the data. Be sure to specify the Treasure Data database and table where the data should be stored.  Treasure Data also recommends specifying the --time-column option because Treasure Data’s storage is partitioned by time (see [data partitioning](https://docs.treasuredata.com/smart/project-product-documentation/data-partitioning-in-treasure-data)). If this option is not provided, the data connector chooses the first long or timestamp column as the partitioning time. The type of the column specified by --time-column must be either of long and timestamp type. If your data doesn’t have a time column, you can add a time column by using the *add_time* filter option. For more details see the documentation for the [add_time Filter Function](https://docs.treasuredata.com/smart/project-product-documentation/add_time-filter-function). ``` $ td connector:issue load.yml --database td_sample_db --table td_sample_table \--time-column created_at ``` The connector:issue command assumes that you have already created a database(td_sample_db) and a table(td_sample_table). If the database or the table does not exist in TD, this command fails. Create the database and table manually or use --auto-create-table option with td connector:issue command to auto-create the database and table. ``` $ td connector:issue load.yml --database td_sample_db --table td_sample_table--time-column created_at --auto-create-table ``` The data connector does not sort records on the server side. To use time-based partitioning effectively, sort records in files beforehand. If you have a field called time, you don’t have to specify the --time-column option. ``` $ td connector:issue load.yml --database td_sample_db --table td_sample_table ``` ### Import Modes Specify the file import mode in the out: section of the load.yml file. The out: section controls how data is imported into a Treasure Data table. For example, you may choose to append data or replace data in an existing table. | **Mode** | **Description** | **Examples** | | --- | --- | --- | | Append | Records are appended to the target table. | in:   ... out:   mode: append | | AlwaysReplace | Replaces data in the target table.Any manual schema changes made to the target table remain intact. | in:   ... out:   mode: replace | | Replace on new data | Replaces data in the target table only when there is new data to import. | in:   ... out:   mode: replace_on_new_data | ### Scheduling Executions You can schedule periodic data connector execution for incremental file import. The Treasure Data scheduler is optimized to ensure high availability. For the scheduled import, you can import all files that match the specified prefix and one of these conditions: - If use_modified_time is disabled, the last path is saved for the next execution. On the second and subsequent runs, the integration only imports files that come after the last path in alphabetical order. - Otherwise, the time that the job is executed is saved for the next execution. On the second and subsequent runs, the connector only imports files that were modified after that execution time in alphabetical order. ### Create a Schedule Using the TD Toolbelt A new schedule can be created using the td connector:create command. ``` $ td connector:create daily_import "10 0 * * *" \td_sample_db td_sample_table load.yml ``` Treasure Data also recommends specifying the --time-column option because Treasure Data’s storage is partitioned by time (see [data partitioning](https://docs.treasuredata.com/smart/project-product-documentation/data-partitioning-in-treasure-data)). ``` $ td connector:create daily_import "10 0 * * *" \td_sample_db td_sample_table load.yml \--time-column created_at ``` The cron parameter also accepts three special options: @hourly, @daily, and @monthly. By default, the schedule is set up in the UTC timezone. You can set the schedule in a timezone using -t or --timezone option.  The --timezone option supports only extended timezone formats like Asia/Tokyo, America/Los_Angeles, etc. Timezone abbreviations like PST, CST are not supported and might lead to unexpected schedules.