Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Install ‘td’ Command v0.11.9 or Later

You can install the newest Treasure Data Toolbelt.

Code Block
linenumberstrue
$ td --version
0.15.3


Create Configuration File

Prepare configuration file (for eg: load.yml) as shown in the following example, with your Salesforce Marketing Cloud account access information.

...

  • client_id: Salesforce Marketing Cloud client id.

  • client_secret: Salesforce Marketing Cloud client secret.

  • target: Salesforce Marketing Cloud entity object to be imported.


Optionally Preview Data to Import

You can preview data to be imported using the command td connector:preview.

Code Block
linenumberstrue
$ td connector:preview load.yml
+-----------------+---------------------+--------------------+----
| id:long         | name:string         | description:string | ...
+-----------------+---------------------+--------------------+----
| 42023           | "Hello"             | apps               |
| 42045           | "World"             | apps               |
+-----------------+---------------------+--------------------+----


Execute Load Job

Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data are stored.

...

You can assign Time Format column to the "Partitioning Key" by "--time-column" option.


Scheduled Execution

You can schedule periodic data connector execution for periodic Salesforce Marketing Cloud import. We configure our scheduler carefully to ensure high availability. By using this feature, you no longer need a cron daemon on your local data center.


Create the Schedule

A new schedule can be created using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required.

...

By default, the schedule is setup in the UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option only supports extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules.


List the Schedules

You can see the list of currently scheduled entries by td connector:list.

Code Block
linenumberstrue
$ td connector:list
+-----------------------------------------+--------------+----------+-------+--------------+-----------------+----------------------------------------------+
| Name                                    | Cron         | Timezone | Delay | Database     | Table           | Config                                       |
+-----------------------------------------+--------------+----------+-------+--------------+-----------------+----------------------------------------------+
| daily_salesforce_marketing_cloud_import | 10 0 * * *   | UTC      | 0     | td_sample_db | td_sample_table | {"type"=>"salesforce_marketing_cloud_v2", ... } |
+-----------------------------------------+--------------+----------+-------+--------------+-----------------+----------------------------------------------+


Show the Setting and History of Schedules

td connector:show shows the execution setting of a schedule entry.

...

Code Block
linenumberstrue
% td connector:history daily_salesforce_marketing_cloud_import
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-18 00:10:05 +0000 | 160      |
| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-17 00:10:07 +0000 | 161      |
| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-16 00:10:03 +0000 | 152      |
| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-15 00:10:04 +0000 | 163      |
| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-14 00:10:04 +0000 | 164      |
| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-13 00:10:04 +0000 | 155      |
| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-12 00:10:05 +0000 | 156      |
| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-11 00:10:04 +0000 | 157      |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
8 rows in set


Delete the Schedule

td connector:delete removes the schedule.

Code Block
linenumberstrue
$ td connector:delete daily_salesforce_marketing_cloud_import


Incremental Loading

For Data Extensions

Treasure Data supports incremental loading for Data Extensions that have a date field.

...

If incremental: false is set, The data connector loads all records for the target specified. This is one-time activity.

For Email Events

Treasure Data supports incremental loading for Email Events based on their event date.

...

If incremental: false is set, The data connector loads all records for the target specified. This is a one-time activity.

Modes for the Out Plugin

Excerpt Include
Import Modes for the Out Section of the Load.yml File
Import Modes for the Out Section of the Load.yml File
nopaneltrue

Sample Configuration for Each Target

For Campaign target using Legacy Package

...

Parameters

Description

Default value

type

must be salesforce_marketing_cloud_v2


client_id

Salesforce Marketing Cloud Client ID


client_secret

Salesforce Marketing Cloud Client Secret


auth_type

Package Type

This is a enum (v1, v2) where v1 is Legacy package and v2 is Enhanced Package

v1

tenant_auth_uri

Tenant Authentication Base URI (only enter this option if auth_type = v1)

This will require since is required from September 2022 as Marketing Cloud is going to deprecate the legacy endponts [https://*.exacttargetapis.com].

For more information please check : https://help.salesforce.com/s/articleView?id=000356497, https://help.salesforce.com/s/articleView?id=000356498


auth_uri


Authentication Base URI (only enter this option if auth_type = v2)


...

Parameters

Description

Default value

type

must be salesforce_marketing_cloud_v2


client_id

Salesforce Marketing Cloud Client ID


client_secret

Salesforce Marketing Cloud Client Secret


auth_type

Package Type

This is a enum (v1, v2) where v1 is Legacy package and v2 is Enhanced Package

v1

tenant_auth_uri

Tenant Authentication Base URI (only enter this option if auth_type = v1)

This will require since is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endponts [https://*.exacttargetapis.com].

For more information please check : https://help.salesforce.com/s/articleView?id=000356497, https://help.salesforce.com/s/articleView?id=000356498


auth_uri


Authentication Base URI (only enter this option if auth_type = v2)


contact_attributes_request_limit

Number of attributes to ingest for each contact in a single request. When number of attributes greater than this value, it is split into multiple requests

100

contact_multiple_requests

Enable plugin to ingest contact detail with so many attributes in multiple requests (This only effects when number of attributes greater than 100 )

false

ignore_attribute_set_names

List of attributes are ignored. Useful when you want to eliminate some unnecessary attributes

null

...

Parameters

Description

Default value

type

must be salesforce_marketing_cloud_v2


client_id

Salesforce Marketing Cloud Client ID


client_secret

Salesforce Marketing Cloud Client Secret


auth_type

Package Type

This is a enum (v1, v2) where v1 is Legacy package and v2 is Enhanced Package

v1

tenant_auth_uri

Tenant Authentication Base URI (only enter this option if auth_type = v1)

This will require since is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endponts [https://*.exacttargetapis.com].

For more information please check : https://help.salesforce.com/s/articleView?id=000356497, https://help.salesforce.com/s/articleView?id=000356498


auth_uri


Authentication Base URI (only enter this option if auth_type = v2)


shared_data_extension

Set this flag to true when you want to ingest the data from shared data extensions

false

incremental

Set this flag to true if you want to get newer data between each schedule only

false

incremental_column_name

Column used for incremental loading. Must be set if you set incremental = true

null

from_date

Set this to ingest data from this date

null

fetch_days

Duration to ingest is from_date to from_date + fetch_days

1

...

Parameters

Description

Default value

type

must be salesforce_marketing_cloud_v2


client_id

Salesforce Marketing Cloud Client ID


client_secret

Salesforce Marketing Cloud Client Secret


auth_type

Package Type

This is a enum (v1, v2) where v1 is Legacy package and v2 is Enhanced Package

v1

tenant_auth_uri

Tenant Authentication Base URI (only enter this option if auth_type = v1)

This will require since is required as of September 2022 as Marketing Cloud is going to deprecate the legacy endponts [https://*.exacttargetapis.com].

For more information please check : https://help.salesforce.com/s/articleView?id=000356497, https://help.salesforce.com/s/articleView?id=000356498


auth_uri


Authentication Base URI (only enter this option if auth_type = v2)


target

must be email_event (in this case)


search_term

Name of email you want to ingest

null

search_mode

Mode for name matching pattern. Must be exact or partial only

exact

email_event_types

List of event types you want to import. Valid options are as follows:

  • value : all

  • value : bounce

  • value : click

  • value : forward

  • value : forwardOptIn

  • value : open

  • value : sent

  • value : survey

  • value : unsub

null

start_time

Set this to ingest data from this time

null

end_time

Set this to ingest data to this time

null

incremental

Set this flag to true if you want to get newer data between each schedule only

false

...