Learn more about Marketo Export Integration.
You can import Marketo data into Treasure Data. Then, in Treasure Data, you can integrate the data with your other data sources. You can also import using the CLI ( see Marketo Import Connection Using CLI).
Bulk extracts source are imported using Marketo Bulk Extract API.
For sample workflows for importing Marketo data, see Treasure Boxes.
- Basic knowledge of Treasure Data.
- There is a daily quota of 500MB Bulk Extract per day. If you reach the limit, we recommend that you reduce the import date range, or contact Marketo account manager and pay for additional space.
- Marketo will retain high-volume activity data for 90 days past the activity date and all other activity data for at least 25 months past the activity date. You can view Marketo's Activities Data Retention Policy document for more details.
If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.
Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/
Access to your API-enabled account is required before you can access the API. The Munchkin Account ID can be retrieved from the Marketo Admin page.
To enable and create a new account for API access:
- Create an API Only User.
- Create an API Only User Role.
- Get the Custom Object API name.
4. Get the Custom Object field name.
Your first step is to create a new authentication with a set of credentials.
- Select Integrations Hub.
- Select Catalog.
3. Search for your Integration in the Catalog; hover your mouse over the icon and select Create Authentication.
4. Ensure that the Credentials tab is selected and then enter credential information for the integration.
New Authentication Fields
| Parameter | Description |
|---|---|
| Marketo Account ID (required) | This is your Marketo Service/Munchkin ID. |
| Marketo Client ID (required) | This is a service-specific client ID. |
| Marketo Client Secret (required) | This is a service specific client secret. |
- Select Continue.
- Enter a name for your authentication and select Done.
- Open TD Console.
- Navigate to Integrations Hub > Authentications.
- Locate your new authentication and select New Source.
The Create Source modal opens.
- Type a name in the Data Transfer field.
2. Select Next.
- From the Source drop-down menu, choose one of the following sources:
- Leads
- Activities
- Campaigns
- Leads by Static List
- Leads by Program
- Programs
- Custom Object
- Program Members
The values to enter differ based on your choice of source:
Leads
Activities
Campaigns
Leads by Static List
Leads by Program
Programs
Custom Object
Program Members
| Parameter | Description |
|---|---|
| Incremental | Bulk extract sources support the incremental import feature through the createdAt filter. The connector calculates the next from_date when running in an incremental mode based on the previous filter. Example: - 1st run from_date: 2018-01-01 to_date: 2018-01-11(generated by plugin) fetch_days: 10 - 2nd run from_date: 2018-01-11 to_date: 2018-01-21 fetch_days: 10Limitation: – Bulk extract source preview shows only mock data. |
| updateAt | updatedAt is used when you want to retrieve data based on the update date for incremental loading. updatedAt without incremental will allow you to get the updated leads within a specific amount of time |
| From Date | Data that has a createdAt or updatedAt filter after the specified date. |
| Fetch Days | The To date is calculated using From day + Fetch days |
| Escape Character | The Marketo CSV file escape character |
| Quote Character | The Marketo CSV file quote character |
| Activity Type IDs | Integer values correspond to the desired activity types. |
| List IDs | Retrieves person records that are members of the given static list. Results include the column listId that contains the lead list ID. Optional. Comma separated List IDs, or leave the field blank to import all lists. |
| Program IDs | Optional. Comma separated program IDs. Or leave the field blank to import all programs. |
| Query By | Query Programs by Tag Type or Date Range. Or, leave blank to import all programs. With Query By: Tag Type, Incremental import is not supported and therefore is omitted. With Query By Date Range and Incremental import is selected, the next run Earliest Updated At and Latest Updated At will be calculated based on previous range, example:- 1st run: Earliest Updated At = 02/01/2018 10:00, Latest Updated At = 02/10/2018 10:00- 2nd run: Earliest Updated At = 02/10/2018 10:01, Latest Updated At = 02/20/2018 10:01 |
| API Name | The API name of the custom object |
| Fields | Optional. Comma-separated API name of fields of the custom object. |
| Filter Type | Comma-separated list of fields API name needs to return. If unset marketoGuid, dedupe Fields, updatedAt, createdAt will be returnedInput filter values as text (comma separated), or by range (From Value to To Value). |
| Input as Text | Input filter values as text (comma separated), or by range (From Value to To Value). |
| Filter Values | Comma-separated list of field values to match. |
| From Value | Filter start value. Only supports integers. |
| To Value | Filter end value (Optional). Only supports integers. Only records with a value greater than "From Value" will be returned if not set. The job will stop if no record is found in 300 consecutive values. |
| Program IDs | Optional. Comma-separated program IDs, or leave the field blank to import members of all programs. |
- Select Next.
- Optionally, edit the data settings or skip this page.
| Parameters | Description |
|---|---|
| LEAD INCLUDED FIELDS | Add a list of Lead fields to be included in data import, which only affects the Lead family target |
| Interval limit in milliseconds | Time to wait for the next call if the request reaches Marketo concurrent limit |
| Maximum retries | Maximum times to retry Marketo request when an error occurs. |
| Batch size | Marketo REST API Batch size (default 300). |
| Max return | Max records to return in a single request. The program endpoint uses offset for paging (default 200). |
| Bulk job timeout in seconds | Total time wait for bulk extract before failing the job |
| Bulk job polling interval in seconds | Interval to poll job status |
| Read timeout in milliseconds | Time to wait for Marketo's response |
You can see a preview of your data before running the import by selecting Generate Preview. Data preview is optional and you can safely skip to the next page of the dialog if you choose to.
- Select Next. The Data Preview page opens.
- If you want to preview your data, select Generate Preview.
- Verify the data.
For data placement, select the target database and table where you want your data placed and indicate how often the import should run.
Select Next. Under Storage, you will create a new or select an existing database and create a new or select an existing table for where you want to place the imported data.
Select a Database > Select an existing or Create New Database.
Optionally, type a database name.
Select a Table> Select an existing or Create New Table.
Optionally, type a table name.
Choose the method for importing the data.
- Append (default)-Data import results are appended to the table. If the table does not exist, it will be created.
- Always Replace-Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created.
- Replace on New Data-Only replace the entire content of an existing table with the result output when there is new data.
Select the Timestamp-based Partition Key column. If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter.
Select the Timezone for your data storage.
Under Schedule, you can choose when and how often you want to run this query.
- Select Off.
- Select Scheduling Timezone.
- Select Create & Run Now.
- Select On.
- Select the Schedule. The UI provides these four options: @hourly, @daily and @monthly or custom cron.
- You can also select Delay Transfer and add a delay of execution time.
- Select Scheduling Timezone.
- Select Create & Run Now.
After your transfer has run, you can see the results of your transfer in Data Workbench > Databases.