Scheduled execution supports additional configuration parameters that control the behavior of the data connector during its periodic attempts to fetch data from the integration source:
incrementalThis configuration is used to control the load mode, which governs how the data connector fetches data from the integration based on one of the native timestamp or numeric field associated with each objectincremental_columnnThis configuration is used to define a based column to imported into Treasure Data. You can define only one column for this field. Suggested value is created, createdTimestamp, updated, updatedTimestamp- Timezone abbreviations like PST, CST are not supported and may lead to unexpected schedules.
- The --timezone option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc.
Here’s an example of a load file using incremental mode
in:
type: intg_type
data_center: US1
authentication_mode: key_secret
application_key: your_application_user_key
secret_key: your_application_secret_key
api_key: your_api_key
data_source: account
batch_size: 1000
query: SELECT * FROM table_name
incremental: true
incremental_column: created
filters:
- type: add_time
from_value:
mode: upload_time
to_column:
name: timeA new schedule can be created using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the data connector configuration file are required.
The cron parameter accepts these options: @hourly, @daily and @monthly.
By default, the schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option.
For example, you can create a scheduled job using the command td connector:create to run daily:
td connector:create connector_name @daily connector_database connector_table load.yml