Data Connector for Amplitude

The Data Connector for Amplitude enables import of data on your Amplitude account.

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data

Step 0: Install ‘td’ command v0.11.9 or later

Install the newest Treasure Data Toolbelt.

$ td --version
0.11.10

Step 1: Create seed file

First, please prepare seed.yml as below, with your credentials.

in:
  type: amplitude
  api_key: "<YOUR_API_KEY>"
  secret_key: "<YOUR_SECRET_KEY>"
  start_time: "20160901T03" # UTC Timezone. The format is yyyymmddThh. "T" is a static character.

Step 2: Guessing

Second, please use connector:guess. This command automatically reads the target data, and intelligently guesses the data format.

$ td connector:guess seed.yml -o load.yml

If you open up load.yml, you’ll see guessed file transformed like below.

in: {type: amplitude, api_key: API_KEY, secret_key: SECRET_KEY,
  start_time: 20160901T03}
out: {}
exec: {}
filters:
- type: rename
  rules:
  - rule: upper_to_lower
  - rule: character_types
    pass_types: ["a-z", "0-9"]
    pass_characters: "_"
    replace: "_"
  - rule: first_character_types
    pass_types: ["a-z"]
    pass_characters: "_"
    prefix: "_"
  - rule: unique_number_suffix
    max_length: 128
- type: add_time
  to_column: {name: time}
  from_value: {mode: upload_time}

For more details on the rename filter, see rename filter plugin for Data Connector.

For more details on available out modes, see Appendix.

Then, you can preview how the system will parse the file by using the preview command.

$ td connector:preview load.yml
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
| app:long | amplitude_id:long | user_id:string       | device_id:string                       | event_time:timestamp | server_upload_time:timestamp  | client_event_time:timestamp | client_upload_time:timestamp  | event_id:long | session_id:long | event_type:string | amplitude_event_type:string | version_name:string | _schema:long | adid:string | groups:json | idfa:string | library:string | processed_time:timestamp         | user_creation_time:timestamp | platform:string | os_name:string | os_version:string | device_brand:string | device_manufacturer:string | device_model:string | device_carrier:string | device_type:string | device_family:string | location_lat:double | location_lng:double | country:string  | language:string | city:string | region:string | dma:string | revenue:double | ip_address:string | paying:boolean | start_version:string | event_properties:json | user_properties:json      | data:json                | uuid:string                            | _insert_id:string | time:timestamp                |
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 04:25:31.405 UTC" | nil                         | "2016-09-13 04:25:31.405 UTC" | 823811538     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 04:25:32.791225 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{\"first_event\":true}" | "1bc49492-796a-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 04:25:36.952 UTC" | nil                         | "2016-09-13 04:25:36.952 UTC" | 38872708      | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 04:25:38.171848 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "1f635fde-796a-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 05:00:03.679 UTC" | nil                         | "2016-09-13 05:00:03.679 UTC" | 689967922     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 05:00:04.692970 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "ef211fbe-796e-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 06:00:25.511 UTC" | nil                         | "2016-09-13 06:00:25.511 UTC" | 44814028      | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 06:00:26.509410 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "5da3baf2-7977-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 06:00:25.541 UTC" | nil                         | "2016-09-13 06:00:25.541 UTC" | 847463167     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 06:00:26.735517 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "5e0badba-7977-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 07:00:00.437 UTC" | nil                         | "2016-09-13 07:00:00.437 UTC" | 808687647     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 07:00:01.389360 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "b07c57b8-797f-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 07:00:00.589 UTC" | nil                         | "2016-09-13 07:00:00.589 UTC" | 145913611     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 07:00:01.178515 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "b091fe4c-797f-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 08:00:26.193 UTC" | nil                         | "2016-09-13 08:00:26.193 UTC" | 118354173     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 08:00:26.888901 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "21a9a762-7988-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 08:00:26.297 UTC" | nil                         | "2016-09-13 08:00:26.297 UTC" | 177419860     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 08:00:27.323531 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "21ebf04a-7988-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 09:00:00.890 UTC" | nil                         | "2016-09-13 09:00:00.890 UTC" | 256548632     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 09:00:01.668570 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "7409a27a-7990-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 09:00:00.982 UTC" | nil                         | "2016-09-13 09:00:00.982 UTC" | 496536219     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 09:00:01.650288 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "746275a8-7990-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 10:16:46.350 UTC" | nil                         | "2016-09-13 10:16:46.350 UTC" | 958097131     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 10:16:46.663451 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "2d2fb2bc-799b-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 10:16:46.393 UTC" | nil                         | "2016-09-13 10:16:46.393 UTC" | 474656866     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 10:16:47.669048 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "2dc7129c-799b-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 14:25:43.820 UTC" | nil                         | "2016-09-13 14:25:43.820 UTC" | 562771389     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 14:25:44.723185 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "f4f3f0ca-79bd-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 14:25:43.825 UTC" | nil                         | "2016-09-13 14:25:43.825 UTC" | 714909881     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 14:25:45.332413 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "f5472bb4-79bd-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 16:25:52.992 UTC" | nil                         | "2016-09-13 16:25:52.992 UTC" | 966142739     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 16:25:54.925309 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "be8c64a2-79ce-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 16:25:52.980 UTC" | nil                         | "2016-09-13 16:25:52.980 UTC" | 136550534     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 16:25:55.159777 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "beb0e872-79ce-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 18:26:06.237 UTC" | nil                         | "2016-09-13 18:26:06.237 UTC" | 911332926     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 18:26:07.308899 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "897c0d24-79df-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 18:26:06.582 UTC" | nil                         | "2016-09-13 18:26:06.582 UTC" | 637365591     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 18:26:07.932679 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "89d72740-79df-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 21:26:14.886 UTC" | nil                         | "2016-09-13 21:26:14.886 UTC" | 325299849     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 21:26:15.879945 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "b3e28336-79f8-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 21:26:14.881 UTC" | nil                         | "2016-09-13 21:26:14.881 UTC" | 275591044     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 21:26:15.902873 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "b3e89ca8-79f8-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-14 01:29:27.029 UTC" | nil                         | "2016-09-14 01:29:27.029 UTC" | 335311243     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 01:29:28.287592 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "ada09ba8-7a1a-11e6-9f4f-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-14 01:29:26.988 UTC" | nil                         | "2016-09-14 01:29:26.988 UTC" | 633523172     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 01:29:28.350201 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "ada9cae8-7a1a-11e6-bead-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-14 02:00:26.207 UTC" | nil                         | "2016-09-14 02:00:26.207 UTC" | 449936451     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 02:00:26.662638 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "0157e478-7a1f-11e6-bead-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
24 rows in set
Update load.yml and use 'td connector:preview load.yml' to preview again.
Use 'td connector:issue load.yml' to run Server-side bulk load.

Step 3: Execute Load Job

Finally, submit the load job. It may take a couple of hours depending on the size of the data.

Untitled-3
Amplitude connector provides `time` column by automatically, but also you may specify it as any timestamp column such as `--time-column server_upload_time`.
$ td connector:issue load.yml --database td_sample_db --table td_sample_table

Scheduled execution

You can schedule periodic Data Connector execution for incremental Amplitude. We take great care in distributing and operating our scheduler in order to achieve high availability. By using this feature, you no longer need a cron daemon on your local datacenter.

For the scheduled import at first run, the Data Connector for Amplitude imports all of your ad data.

On the second and subsequent runs, it will only imports files that comes after the last execution.

Create the schedule

A new schedule can be created using the td connector:create command. The following are required: the name of the schedule, the cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file.

$ td connector:create \
    daily_import \
    "10 0 * * *" \
    td_sample_db \
    td_sample_table \
    load.yml
Untitled-3
The `cron` parameter also accepts three special options: `@hourly`, `@daily` and `@monthly`.
Untitled-3
By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. Please note that `--timezone` option only supports extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules.

List the Schedules

You can see the list of currently scheduled entries by running the command td connector:list.

$ td connector:list
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+
| Name         | Cron       | Timezone | Delay | Database     | Table           | Config                     |
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+
| daily_import | 10 0 * * * | UTC      | 0     | td_sample_db | td_sample_table | {"in"=>{"type"=>"amp", ... |
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+

Show the Settings and Schedule History

td connector:show shows the execution settings of a schedule entry.

% td connector:show daily_import
Name     : daily_import
Cron     : 10 0 * * *
Timezone : UTC
Delay    : 0
Database : td_sample_db
Table    : td_sample_table
Config
---
in:
  type: amplitude
  api_key: "<YOUR_API_KEY>"
  secret_key: "<YOUR_SECRET_KEY>"
  start_time: "20160901T03" # UTC Timezone. The format is yyyymmddThh. "T" is a static character.
out:
  mode: append

td connector:history shows the execution history of a schedule entry. To investigate the results of each individual run, please use td job <jobid>.

% td connector:history daily_import
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-18 00:10:05 +0000 | 160      |
| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-17 00:10:07 +0000 | 161      |
| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-16 00:10:03 +0000 | 152      |
| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-15 00:10:04 +0000 | 163      |
| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-14 00:10:04 +0000 | 164      |
| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-13 00:10:04 +0000 | 155      |
| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-12 00:10:05 +0000 | 156      |
| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-11 00:10:04 +0000 | 157      |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
8 rows in set

Delete the Schedule

td connector:delete will remove the schedule.

$ td connector:delete daily_import

Configuration

See the following table for more details on available in modes.

Option name Description Type Required? Default value
api_key API key string yes N/A
secret_key API Secret key string yes N/A
start_time First hour included in data series, formatted YYYYMMDDTHH (e.g. ‘20150201T05’). This is UTC timezone. string yes N/A
end_time Last hour included in data series, formatted YYYYMMDDTHH (e.g. ‘20150203T20’). This is UTC timezone. string no processed time
incremental true for “mode: append”, false for “mode: replace” (See below). bool no true

Appendix

A) Modes for out plugin

You can specify file import mode in out section of seed.yml.

append (default)

This is the default mode and records are appended to the target table.

in:
  ...
out:
  mode: append

replace (In td 0.11.10 and later)

This mode replaces data in the target table. Please note that any manual schema changes made to the target table will remain intact with this mode.

in:
  ...
out:
  mode: replace

Last modified: Feb 24 2017 09:27:52 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.