Incremental Loading for Data Extensions

Treasure Data supports incremental loading for Data Extensions that have a date field.

If incremental loading is selected, the integration loads records according to the range specified by the from_date and the fetch_days for the specified date field.

The following list provides details about all available options:

  • target: Target object. For example, a Customer or Invoice that accepts a search operation. (string, required)

  • email: Email address. (string, required)

  • password: Password. (string, required)

  • account: Account ID. (string, required)

  • role: Role ID. (string, default: nil)

  • sandbox: Use sandbox environment if true. (bool, default: false)

  • from_datetime: Fetch data after this time. (string, default: nil)

Example

For example, you set up an incremental job with
Start Time = 2020-11-01T01:43:47.900Z and End Time = 2020-12-01T01:43:47.900Z

  • After the first run, the next start time will be calculated base on the precision setting.

    • HOURLY next start time will be 2020-12-01T02:43:47.900Z

    • DAILY next start time will be 2020-12-02T01:43:47.900Z

    • MONTHLY next start time will be 2021-01-01T01:43:47.900Z

  • And the End Time will be the current execution time.


  • No labels