Scheduled execution supports additional configuration parameters that control the behavior of the data connector during its periodic attempts to fetch data from the integration source:

  • incremental This configuration is used to control the load mode, which governs how the data connector fetches data from the integration based on one of the native timestamp or numeric field associated with each object

  • incremental_columnn This configuration is used to define a based column to imported into Treasure Data. You can define only one column for this field. Suggested value is created, createdTimestamp, updated, updatedTimestamp

  • Timezone abbreviations like PST, CST are not supported and may lead to unexpected schedules.

  • The `--timezone` option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc.

Here’s an example of a load file using incremental mode

in: 
  type: <intg_type>
  data_center: US1
  authentication_mode: key_secret
  application_key: your_application_user_key 
  secret_key: your_application_secret_key 
  api_key: your_api_key
  data_source: account 
  batch_size: 1000
  query: SELECT * FROM <table_name> 
  incremental: true
  incremental_column: created
filters:
- type: add_time
  from_value: 
    mode: upload_time 
  to_column: 
    name: time

 Create the Schedule

A new schedule can be created using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the data connector configuration file are required.

The `cron` parameter accepts these options: `@hourly`, `@daily` and `@monthly`.

By default, the schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option.

For example, you can create a scheduled job using the command td connector:create to run daily:

td connector:create connector_name @daily connector_database connector_table load.yml

 

  

 

  • No labels