Complete the following steps to migrate from the legacy Salesforce Legacy data connector to the new Salesforce connector. The legacy data connector uses only REST API to import data. The new Salesforce data connector enables you to use Bulk import and REST API.
Characteristics of Ingested Data
When migrating data from one place or version, it is worth being aware of how that data might be transformed. The following sections outline some important characteristics to be aware of.
Campaign
You can ingest more than 50 assets.
Column | Old Data Type | New Data Type |
---|---|---|
createdDate | string | timestamp |
modifiedDate | string | timestamp |
Campaign Assets
Column | Old Data Type | New Data Type |
---|---|---|
createdDate | string | ISO 8601 string |
Other date time values are converted to UTC.
Contact
Ingestion is limited to:
root and system defined data
one-to-one and one-to-many relationships
one-to-one relationships are saved as a single JSON
one-to-many relationships are saved as a JSON array
Other attributes must be ingested using the Treasure Data ingestion feature.
Contact attributes are collected for root and system, you are not able to limit ingested attributes.
The number of records per page used the default value of 2000.
Data Extension
Ingestion is limited to:
one data extension at a time
Old Column Name | New Column Name |
---|---|
<data-extension>-<column-name> | <column-name> |
Column | Old Data Type | New Data Type | Format of Data |
---|---|---|---|
<any-datetime> | string | timestamp | UTC |
TD generated properties have an underscore prefix of “_” so that they can be easily identified.
The number of records per page used the default value of 2500.
Email Event
Ingestion excludes subscribers associated with an event.
Create a New Salesforce V2 Connector
Go to Treasure Data Catalog, then search and select Salesforce v2.
In the dialog box, enter the values that you enter in your legacy Salesforce connector.
Salesforce v2 connector requires that you remove unnecessary letters from Login URL parameter. For example, instead of https://login.salesforce.com/?locale=jp , use https://login.salesforce.com/ .
Enter your username (your email) and password, as well as your Client ID, Client Secret and Security Token.
Save Settings and Run the Legacy Salesforce Data Connector One Last Time
You can save your legacy setting from TD Console or from the CLI.
Using TD Console
Save the Settings of Your Scheduled Legacy Salesforce Connector and Run a Final Import
Go to Integration Hub > Sources. Search for your scheduled Salesforce source, select the source and select Edit.
In the dialog box, copy the settings to use later:
Also copy any advanced settings:
Next, you configure one final run with the legacy data connector to create a temporary table against which you can run a config-diff. You use the diff to identify and confirm the latest data imported into Treasure Data.
Before running the final import with the legacy connector, make sure that you change the schedule to one run only:
After the job is complete, look at and copy config_diff in job query information somewhere to use later.
Create New Salesforce V2 Source
Go to Integration Hub > Authentication. Search for new Salesforce v2 connection that you created:
Select the New Source. Fill in all basic settings and advanced settings that you copied in the preceding steps. Then, if you want the new source to continue ingesting from the point where the legacy connector left, fill in the Last Record field with the config_diff information that you copied in the previous job.
After completing the settings, choose the database and table job to populate data into, then schedule the job and provide a name for your new data connector. Select Save and then run the new data connector.
Using CLI and Workflow
Update in: type in your yml configuration from sfdc to sfdc_v2.
For example, your existing workflow configuration might look something like this:
in: type: sfdc username: ${secret:sfdc.username} password: ${secret:sfdc.password} client_id: ${secret:sfdc.client_id} client_secret: ${secret:sfdc.client_secret} security_token: ${secret:sfdc.security_token} login_url: ${secret:sfdc.login_url} target: Lead out: {} exec: {} filters: []
Your new workflow configuration would look like this:
in: type: sfdc_v2 username: ${secret:sfdc.username} password: ${secret:sfdc.password} client_id: ${secret:sfdc.client_id} client_secret: ${secret:sfdc.client_secret} security_token: ${secret:sfdc.security_token} login_url: ${secret:sfdc.login_url} target: Lead out: {} exec: {} filters: []
For Result Output
The SFDC connection is shared between data connector and result output, although there is nothing change in result output, if you use either of those, you should upgrade it too.
Using TD Console
Save the Settings of Legacy Export Connector
Go to TD Console. Go to Query Editor. Open the Query that uses SFDC for its connection.
Select the SFDC connector, then copy and save the details of the existing connection to use later.
Select DELETE to remove the Legacy one.
Modify the Existing Query (to Replace the Legacy Connection)
In the query, select Output Results. Next, you are going to set up the SFDC v2 connector by finding and select the SFDC v2 export connector that you created.
In Configuration pane, specify the fields you saved in the previous step, then select Done.
Check Output results to... to verify that you are using the created output connection. Select Save.
It is strongly recommended to create a test target and use it for the first data export to verify that exported data looks as expected and the new export does not corrupt existing data. In your test case, choose an alternate “Object” for your test target. |
Using CLI
Result type protocol needs to update from sfdc to sfdc_v2 for instance from:
sfdc://<username>:<password><security_token>@<hostname>/<object_name>
to:
sfdc_v2://<username>:<password><security_token>@<hostname>/<object_name>
Using Workflow
If you have a workflow that used the SFDC, you can keep your result settings the same, but need to update result_connection to the new connection_name.
An example of old workflow result output settings is as follows:
+td-result-output-sfdc: td>: queries/sample.sql database: sample_datasets result_connection: your_old_connection_name result_settings: object: object_name mode: append concurrency_mode: parallel retry: 2 split_records: 10000
An example of new workflow result output settings is as follows:
+td-result-output-sfdc: td>: queries/sample.sql database: sample_datasets result_connection: your_new_connection_name result_settings: object: object_name mode: append concurrency_mode: parallel retry: 2 split_records: 10000