# Salesforce Marketing Cloud Import Integration CLI V2 Treasure Data can publish user segments into Salesforce Marketing Cloud, and enable you to send personalized emails to your customers. You can run data-driven email campaigns by using your first-party data from Web, Mobile, CRM, and other data sources. This topic contains information on how to accomplish this using the CLI. ## Install ‘td’ Command v0.11.9 or Later You can install the newest [Treasure Data Toolbelt](https://toolbelt.treasuredata.com/). ``` $ td --version 0.15.3 ``` ## Create Configuration File Prepare configuration file (for eg: `load.yml`) as shown in the following example, with your Salesforce Marketing Cloud account access information. ```yaml in: type: salesforce_marketing_cloud_v2 client_id: client_secret: auth_type: v2 auth_uri: account_id: target: data_extension_names: shared_data_extension: false ensure_latest_data: false incremental: false maximum_retries: 7 initial_retry_interval_millis: 1000 maximum_retry_interval_millis: 120000 filters: - type: add_time to_column: name: time type: timestamp from_value: mode: upload_time - type: rename rules: - rule: upper_to_lower - rule: character_types pass_types: [ "a-z", "0-9" ] pass_characters: "_" replace: "_" out: type: td apikey: endpoint: database: table: time_column: time mode: replace default_timestamp_format: '%d/%m/%Y' ``` This example dumps Salesforce Marketing Cloud `Campaign` Data Source: - `client_id`: Salesforce Marketing Cloud client id. - `client_secret`: Salesforce Marketing Cloud client secret. - `target`: Salesforce Marketing Cloud entity object to be imported. ## Optionally Preview Data to Import You can preview data to be imported using the command `td connector:preview`. ``` $ td connector:preview load.yml ``` ## Execute Load Job Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data are stored. It is recommended to specify `--time-column` option, since Treasure Data’s storage is partitioned by time (see also [data partitioning](https://docs.treasuredata.com/smart/project-product-documentation/data-partitioning-in-treasure-data)). If the option is not given, the data connector selects the first `long` or `timestamp` column as the partitioning time. The type of the column specified by `--time-column` must be either of `long` and `timestamp` type. If your data doesn’t have a time column you can add it using `add_time` filter option. More details at [add_time filter plugin](https://docs.treasuredata.com/smart/project-product-documentation/add_time-filter-function). ``` $ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column modifieddate ``` The preceding command assumes you have already created *database(td_sample_db)* and *table(td_sample_table)*. If the database or the table does not exist in TD this command will not succeed. Therefore, [create the database and table manually](https://docs.treasuredata.com/smart/project-product-documentation/data-management) or use `--auto-create-table` option with `td connector:issue` command to auto create the database and table: ``` $ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column modifieddate --auto-create-table ``` You can assign Time Format column to the "Partitioning Key" by "--time-column" option. ## Scheduling Your Execution You can schedule periodic data connector execution for periodic Salesforce Marketing Cloud import. We configure our scheduler carefully to ensure high availability. By using this feature, you no longer need a `cron` daemon on your local data center. ### Create the Schedule A new schedule can be created using the `td connector:create` command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required. ```bash $ td connector:create \ daily_salesforce_marketing_cloud_import \ "10 0 * * *" \ td_sample_db \ td_sample_table \ load.yml ``` The `cron` parameter also accepts these three options: `@hourly`, `@daily` and `@monthly`. | By default, the schedule is setup in the UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option only supports extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules. ### Setting up Incremental Loading #### For Data Extensions Treasure Data supports incremental loading for **Data Extensions** that have a date field. If `incremental: true` is set, the data connector loads records according to the range specified by the `from_date` and the `fetch_days` for the specified date field. For example: ```yaml incremental_column_name: mydatefield from_date: "2016-09-01T00:00:00.000Z" fetch_days: 2 ``` - 1st iteration: The data connector fetches records from **Sep 01 00:00:00 UTC 2016** to **Sep 03 00:00:00 UTC 2016** - 2nd iteration: The data connector fetches records for the next available 2 day period, from **Sep 03 00:00:00 UTC 2016** until **Sep 05 00:00:00 UTC 2016**. This process repeats for each successive iteration. - When the increment includes the present date, additional records are fetched as each complete time period becomes available. If `incremental: false` is set, The data connector loads all records for the target specified. This is one-time activity. #### For Email Events Treasure Data supports incremental loading for **Email Events** based on their event date. If `incremental: true` is set - 1st iteration: the connector loads the data from specified Start Time (from all time if not set) to specified End Time (to the time the job was scheduled to run if not set) - 2nd iteration: the connector loads the data from the previous End Time to the time job is scheduled If `incremental: false` is set, The data connector loads all records for the target specified. This is a one-time activity. ## Sample Configurations ### Campaign For `Campaign` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: campaign out: ... ``` For `Campaign` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: campaign out: ... ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required from September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | ### Contact For `Contact` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2   client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: contact contact_page_size: 50 contact_attributes_request_limit: 100 contact_multiple_requests: true out: ... ``` For `Contact` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2   client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: contact contact_page_size: 50 contact_attributes_request_limit: 100 contact_multiple_requests: true out: ... ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | | contact_attributes_request_limit | Number of attributes to ingest for each contact in a single request. When number of attributes greater than this value, it is split into multiple requests | 100 | | contact_multiple_requests | Enable plugin to ingest contact detail with so many attributes in multiple requests (This only effects when number of attributes greater than 100 ) | false | | contact_multiple_requests | Enable plugin to ingest contact detail with so many attributes in multiple requests (This only effects when number of attributes greater than 100 ) | false | ### Data Extension For `Data Extension` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: data_extension   data_extension_name: data_extension_1 incremental: true shared_data_extension: true incremental_column_name: date from_date: "2016-09-01T00:00:00.000Z" fetch_days: 1 out: type: td apikey: td_api_key endpoint: td_endpoint database: database table: table time_column: time mode: replace default_timestamp_format: '%d/%m/%Y' ``` For `Data Extension` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: data_extension   data_extension_name: data_extension_1 log_debug_info: false incremental: true shared_data_extension: true incremental_column_name: date from_date: "2016-09-01T00:00:00.000Z" fetch_days: 1 out: type: td apikey: endpoint: database: table:
time_column: time mode: replace default_timestamp_format: '%d/%m/%Y' ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | | shared_data_extension | Set this flag to true when you want to ingest the data from shared data extensions | false | | ensure_latest_data | If enabled, job will be failed when data lag (ingested versus current in SFMC) over 2500 records | false | | log_debug_info | When enabled, detailed debug information, including the request ID, will be logged (default: false) | false | | incremental | Set this flag to true if you want to get newer data between each schedule only | false | | incremental_column_name | Column used for incremental loading. Must be set if you set incremental = true | null | | from_date | Set this to ingest data from this date | null | | fetch_days | Duration to ingest is from_date to from_date + fetch_days | 1 | | fetch_days | Duration to ingest is from_date to from_date + fetch_days | 1 | ### Email Event For `Email Event` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: email_event log_debug_info: false search_term: "email name" search_mode: "exact" email_event_types: - value: all start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true out: ... ``` For `Email Event` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: email_event log_debug_info: false search_term: "email name" search_mode: "exact" email_event_types: - value: all start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true emails_per_batch: 1 out: ... ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | | target | must be email_event (in this case) | | log_debug_info | When enabled, detailed debug information, including the request ID, will be logged (default: false) | false | | search_term | Name of email you want to ingest | null | | search_mode | Mode for name matching pattern. Must be **exact** or **partial** only | exact | | email_event_types | List of event types you want to import. Valid options are as follows: - value : all - value : bounce - value : click - value : forward - value : forwardOptIn - value : open - value : sent - value : survey - value : unsub | null | | start_time | Set this to ingest data from this time | null | | end_time | Set this to ingest data to this time | null | | incremental | Set this flag to true if you want to get newer data between each schedule only | false | | emails_per_batch | Number of emails per batch. It is used to fine-tune the performance of SFMC in case large volume of events. Acceptable value range from 1 - 10000 | 2500 | | maximum_time_out | Maximum timeout in seconds for a connection. It used to fine-tune the performance of SFMC in case large volume of events. Acceptable value range from 60 - 900 | 300 | | custom_filters | Optional list of filters to narrow down which Send records are retrieved. Each filter requires `field`, `operator`, and `value`. See [Custom Filters](#custom-filters-for-email_event-target) for details. | null | ### Custom Filters for `email_event` Target You can use `custom_filters` to narrow down which `Send` (email campaign job) records are retrieved when using the `email_event` target. Custom filters apply to the SOAP API Retrieve request for `Send` objects, so the events are then fetched only for the matching sends. Supported Target Custom filters are only supported with `target: email_event`. They have no effect on other targets such as `email_event_only`, `campaign`, `contact`, `data_extension`, or `send`. #### Configuration Add the `custom_filters` parameter to the `in:` section of your configuration file. Each filter requires three fields: | Field | Type | Required | Description | | --- | --- | --- | --- | | field | string | Yes | The SFMC object field/property name to filter on (e.g., `EmailName`, `CreatedDate`) | | operator | string | Yes | The filter operator (case-insensitive). See the table below for valid operators. | | value | string | Yes | The filter value. For `BETWEEN` and `IN` operators, specify multiple values as CSV (e.g., `'value1,value2'`). | #### Available Operators | Operator | Description | | --- | --- | | `EQUALS` | Exact match | | `NOT_EQUALS` | Not equal | | `GREATER_THAN` | Greater than | | `LESS_THAN` | Less than | | `GREATER_THAN_OR_EQUAL` | Greater than or equal to | | `LESS_THAN_OR_EQUAL` | Less than or equal to | | `LIKE` | Pattern match | | `BEGINS_WITH` | Starts with the specified value | | `ENDS_WITH` | Ends with the specified value | | `CONTAINS` | Contains the specified value | | `NOT_CONTAINS` | Does not contain the specified value | | `EXISTS_IN_STRING` | Exists in string | | `EXISTS_IN_STRING_AS_A_WORD` | Exists in string as a word | | `NOT_EXISTS_IN_STRING` | Does not exist in string | | `BETWEEN` | Between two values (specify as CSV, e.g., `'2023-01-01,2023-12-31'`) | | `IN` | Matches any of the specified values (specify as CSV, e.g., `'value1,value2,value3'`) | #### How Multiple Filters Are Combined When you specify multiple custom filters, they are combined using **AND** logic. There is no support for OR logic between filters. If you also specify `search_term`, it is treated as an additional filter on the `EmailName` field and combined with your custom filters using AND. Note A filter with a blank `field`, blank `value`, or invalid `operator` is silently dropped rather than causing an error. #### Example The following example retrieves email events only for sends whose `EmailName` contains `"sample emails"` **and** whose `CreatedDate` is greater than `2023-01-15T00:00:00Z`: ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: email_event email_event_types: - value: all incremental: true custom_filters: - field: EmailName operator: CONTAINS value: 'sample emails' - field: CreatedDate operator: GREATER_THAN value: '2023-01-15T00:00:00Z' out: ... ``` ### Email Events Only For `Email Events Only` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: email_event_only log_debug_info: false email_event_types: - value: all start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true out: ... ``` For `Email Events Only` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: email_event_only log_debug_info: false email_event_types: - value: all start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true out: ... ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | | target | must be email_event_only (in this case) | | log_debug_info | When enabled, detailed debug information, including the request ID, will be logged (default: false) | false | | email_event_types | List of event types you want to import. Valid options are as follows: - value : all - value : bounce - value : click - value : forward - value : forwardOptIn - value : open - value : sent - value : survey - value : unsub | null | | start_time | Set this to ingest data from this time | null | | end_time | Set this to ingest data to this time | null | | incremental | Set this flag to true if you want to get newer data between each schedule only | false | | maximum_time_out | Maximum timeout in seconds for a connection. It used to fine-tune the performance of SFMC in case large volume of events. Acceptable value range from 60 - 900 | ### Send For `Send` target using **Legacy Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v1 tenant_auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: send log_debug_info: false start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true out: ... ``` For `Send` target using **Enhanced Package** ```yaml in: type: salesforce_marketing_cloud_v2 client_id: XXXXXX client_secret: XXXXXX auth_type: v2 auth_uri: https://your_tenant_specific_endpoint.auth.marketingcloudapis.com target: send log_debug_info: false start_time: "2019-03-26T07:20:00.000Z" end_time: "2019-03-26T07:53:00.000Z" incremental: true out: ... ``` | Parameters | Description | Default value | | --- | --- | --- | | type | must be salesforce_marketing_cloud_v2 | | client_id | Salesforce Marketing Cloud `Client ID` | | client_secret | Salesforce Marketing Cloud `Client Secret` | | auth_type | Package Type This is a enum (**v1**, **v2**) where **v1** is **Legacy package** and **v2** is **Enhanced Package** | v1 | | tenant_auth_uri | Tenant Authentication Base URI (only enter this option if auth_type = v1) This is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endpoints [https://*.[exacttargetapis.com](http://exacttargetapis.com)]. For more information: [https://help.salesforce.com/s/articleView?id=000356497](https://help.salesforce.com/s/articleView?id=000356497), [https://help.salesforce.com/s/articleView?id=000356498](https://help.salesforce.com/s/articleView?id=000356498) | | auth_uri | Authentication Base URI (only enter this option if auth_type = v2) | | target | must be send (in this case) | | log_debug_info | When enabled, detailed debug information, including the request ID, will be logged (default: false) | false | | start_time | Set this to ingest data from this time | null | | end_time | Set this to ingest data to this time | null | | incremental | Set this flag to true if you want to get newer data between each schedule only | false | | maximum_time_out | Maximum timeout in seconds for a connection. It used to fine-tune the performance of SFMC in case large volume of events. Acceptable value range from 60 - 900 |