Learn more about Amplitude Export Integration.

You can integrate Amplitude with Treasure Data to enhance data, add more data points, and to integrate the data Amplitude collects with all the other parts of your marketing stack.

To view sample workflows for importing data from Amplitude, view Treasure Boxes.

You use this same connector to create and export Amplitude Events. See Amplitude Export Integration.

Integrating Amplitude with Treasure Data makes it easy to:

  • Add new features to Amplitude. As an example, you can use Treasure Data to unify Amplitude behavior data with customer data from Salesforce, creating personally identifiable records with cross-device behavior tracking.

  • Use data collected from Amplitude to make the rest of your marketing stack smarter. For example, you can use Treasure Data and Amplitude to reduce churn by setting up automated triggers to create segments based on usage and feeding them into custom nurture funnels in Marketo.

If you don’t have a Treasure Data account, contact us so we can get you set up.
This topic includes:

How to Connect to Amplitude Using the Treasure Data Console

Connecting to Amplitude using the Treasure Data Console is quick and easy. Alternatively, you can use a CLI to create connections.

In Amplitude

Go to Settings -> Projects and get your API Key and Secret Key, which you need for the next step.

In the Treasure Data Console

Go to Connections -> Sources Catalog and select the Amplitude tile.


Enter your API Key and Secret Key and select Continue.

Give your connection a descriptive name and select Create Connection.

How to Transfer Your Amplitude Data to Treasure Data

Select the Amplitude connection you created in Connections -> My Connections.

Select New Transfer.

Enter the start time in the format YYYYMMDDTHH.

Optionally, you can specify an End Time using the same format. If End Time is not specified the default is the current time of your browser timezone.

Select Next to preview your data.

You’ll see a preview of your data. Select Next.

If you are creating a new database, check Create new database and give your database a name. Do the same with Create new table

Select whether to append records to an existing table or replace your existing table.

If you want to set a different partition key seed than the default key, you can specify a key using the popup menu.

In the When tab, you can specify a one-time transfer, or you can schedule an automated recurring transfer. If you select Once now, select Start Transfer. If you select Repeat… specify your schedule options, then select Schedule Transfer.

After your transfer runs, you can see the results of your transfer in the Databases tab.

How to Connect to Amplitude Using the Command Line

Before you can use the command line to configure a connector, you must install the Treasure Data Toolbelt.

Install the Treasure Data Toolbelt

Install the newest Treasure Data Toolbelt.

$ td --version
0.15.3

Create a Seed File

First, create a file called seed.yml as shown in the following example, with your credentials.

in:
  type: amplitude
  api_key: "<YOUR_API_KEY>"
  secret_key: "<YOUR_SECRET_KEY>"
  start_time: "20160901T03" # UTC Timezone. The format is yyyymmddThh. "T" is a static character.


Run the Guess Fields Command

Run the following command in your terminal:

$ td connector:guess seed.yml -o load.yml

Connector:guess automatically reads the target data and intelligently guesses the data format.

Open the file load.yml, where you’ll see guessed file format definitions including, in some cases, file formats, encodings, column names, and types.

in: {type: amplitude, api_key: API_KEY, secret_key: SECRET_KEY,
  start_time: 20160901T03}
out: {}
exec: {}
filters:
- type: rename
  rules:
  - rule: upper_to_lower
  - rule: character_types
    pass_types: ["a-z", "0-9"]
    pass_characters: "_"
    replace: "_"
  - rule: first_character_types
    pass_types: ["a-z"]
    pass_characters: "_"
    prefix: "_"
  - rule: unique_number_suffix
    max_length: 128
- type: add_time
  to_column: {name: time}
  from_value: {mode: upload_time}

For more details on the rename filter, see the rename filter plugin for Data Connector (needs link).

You can preview how the system will parse the file by using the preview command.

$ td connector:preview load.yml
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
| app:long | amplitude_id:long | user_id:string       | device_id:string                       | event_time:timestamp | server_upload_time:timestamp  | client_event_time:timestamp | client_upload_time:timestamp  | event_id:long | session_id:long | event_type:string | amplitude_event_type:string | version_name:string | _schema:long | adid:string | groups:json | idfa:string | library:string | processed_time:timestamp         | user_creation_time:timestamp | platform:string | os_name:string | os_version:string | device_brand:string | device_manufacturer:string | device_model:string | device_carrier:string | device_type:string | device_family:string | location_lat:double | location_lng:double | country:string  | language:string | city:string | region:string | dma:string | revenue:double | ip_address:string | paying:boolean | start_version:string | event_properties:json | user_properties:json      | data:json                | uuid:string                            | _insert_id:string | time:timestamp                |
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 04:25:31.405 UTC" | nil                         | "2016-09-13 04:25:31.405 UTC" | 823811538     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 04:25:32.791225 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{\"first_event\":true}" | "1bc49492-796a-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 04:25:36.952 UTC" | nil                         | "2016-09-13 04:25:36.952 UTC" | 38872708      | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 04:25:38.171848 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "1f635fde-796a-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 05:00:03.679 UTC" | nil                         | "2016-09-13 05:00:03.679 UTC" | 689967922     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 05:00:04.692970 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "ef211fbe-796e-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 06:00:25.511 UTC" | nil                         | "2016-09-13 06:00:25.511 UTC" | 44814028      | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 06:00:26.509410 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "5da3baf2-7977-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 06:00:25.541 UTC" | nil                         | "2016-09-13 06:00:25.541 UTC" | 847463167     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 06:00:26.735517 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "5e0badba-7977-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 07:00:00.437 UTC" | nil                         | "2016-09-13 07:00:00.437 UTC" | 808687647     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 07:00:01.389360 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "b07c57b8-797f-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 07:00:00.589 UTC" | nil                         | "2016-09-13 07:00:00.589 UTC" | 145913611     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 07:00:01.178515 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "b091fe4c-797f-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 08:00:26.193 UTC" | nil                         | "2016-09-13 08:00:26.193 UTC" | 118354173     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 08:00:26.888901 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "21a9a762-7988-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 08:00:26.297 UTC" | nil                         | "2016-09-13 08:00:26.297 UTC" | 177419860     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 08:00:27.323531 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "21ebf04a-7988-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 09:00:00.890 UTC" | nil                         | "2016-09-13 09:00:00.890 UTC" | 256548632     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 09:00:01.668570 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "7409a27a-7990-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 09:00:00.982 UTC" | nil                         | "2016-09-13 09:00:00.982 UTC" | 496536219     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 09:00:01.650288 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "746275a8-7990-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 10:16:46.350 UTC" | nil                         | "2016-09-13 10:16:46.350 UTC" | 958097131     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 10:16:46.663451 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "2d2fb2bc-799b-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 10:16:46.393 UTC" | nil                         | "2016-09-13 10:16:46.393 UTC" | 474656866     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 10:16:47.669048 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "2dc7129c-799b-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 14:25:43.820 UTC" | nil                         | "2016-09-13 14:25:43.820 UTC" | 562771389     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 14:25:44.723185 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "f4f3f0ca-79bd-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 14:25:43.825 UTC" | nil                         | "2016-09-13 14:25:43.825 UTC" | 714909881     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 14:25:45.332413 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "f5472bb4-79bd-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 16:25:52.992 UTC" | nil                         | "2016-09-13 16:25:52.992 UTC" | 966142739     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 16:25:54.925309 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "be8c64a2-79ce-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 16:25:52.980 UTC" | nil                         | "2016-09-13 16:25:52.980 UTC" | 136550534     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 16:25:55.159777 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "beb0e872-79ce-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 18:26:06.237 UTC" | nil                         | "2016-09-13 18:26:06.237 UTC" | 911332926     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 18:26:07.308899 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "897c0d24-79df-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 18:26:06.582 UTC" | nil                         | "2016-09-13 18:26:06.582 UTC" | 637365591     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 18:26:07.932679 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "89d72740-79df-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-13 21:26:14.886 UTC" | nil                         | "2016-09-13 21:26:14.886 UTC" | 325299849     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 21:26:15.879945 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "b3e28336-79f8-11e6-ae6a-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-13 21:26:14.881 UTC" | nil                         | "2016-09-13 21:26:14.881 UTC" | 275591044     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-13 21:26:15.902873 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "b3e89ca8-79f8-11e6-ba91-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158089806       | "alice@gmail.com"    | "600bfc44-73b1-5e36-bc2f-d230dad9b1fd" | nil                  | "2016-09-14 01:29:27.029 UTC" | nil                         | "2016-09-14 01:29:27.029 UTC" | 335311243     | -1              | "watch_tutorial"  | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 01:29:28.287592 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "United States" | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test A\"}" | "{}"                     | "ada09ba8-7a1a-11e6-9f4f-22000a2589e5" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-14 01:29:26.988 UTC" | nil                         | "2016-09-14 01:29:26.988 UTC" | 633523172     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 01:29:28.350201 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "ada9cae8-7a1a-11e6-bead-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
| 154962   | 10158359360       | "john@gmail.com"     | "b37e7b0d-5474-5618-ae0a-e3d5b36b35f6" | nil                  | "2016-09-14 02:00:26.207 UTC" | nil                         | "2016-09-14 02:00:26.207 UTC" | 449936451     | -1              | "eventtest"       | nil                         | nil                 | 10           | nil         | "{}"        | nil         | "http/1.0"     | "2016-09-14 02:00:26.662638 UTC" | nil                          | nil             | nil            | nil               | nil                 | nil                        | nil                 | nil                   | nil                | nil                  | nil                 | nil                 | "Japan"         | nil             | nil         | nil           | nil        | nil            | "127.0.0.1"       | nil            | nil                  | "{}"                  | "{\"Cohort\":\"Test B\"}" | "{}"                     | "0157e478-7a1f-11e6-bead-22000a5680d9" | nil               | "2016-09-28 08:00:10.137 UTC" |
+----------+-------------------+----------------------+----------------------------------------+----------------------+-------------------------------+-----------------------------+-------------------------------+---------------+-----------------+-------------------+-----------------------------+---------------------+--------------+-------------+-------------+-------------+----------------+----------------------------------+------------------------------+-----------------+----------------+-------------------+---------------------+----------------------------+---------------------+-----------------------+--------------------+----------------------+---------------------+---------------------+-----------------+-----------------+-------------+---------------+------------+----------------+-------------------+----------------+----------------------+-----------------------+---------------------------+--------------------------+----------------------------------------+-------------------+-------------------------------+
24 rows in set
Update load.yml and use 'td connector:preview load.yml' to preview again.
Use 'td connector:issue load.yml' to run Server-side bulk load.


Execute a Load Job

Finally, submit the load job. It may take a couple of hours depending on the size of your data.

Amplitude connector provides a time column automatically, but you can also specify it as any timestamp column such as --time-column server_upload_time.

$ td connector:issue load.yml --database td_sample_db --table td_sample_table

Scheduled Execution

You can also schedule incremental, periodic Data Connector execution from the command line, removing the need for a cron daemon on your local data center.

For the first scheduled import, the Data Connector for Amplitude imports all of your data. On the second and subsequent runs, only newly added files are imported.

Create the Schedule

A new schedule can be created by using the td connector:create command.

The name of the schedule, cron-style schedule, database and table where data will be stored, and the Data Connector configuration file are required.

$ td connector:create \
    daily_import \
    "10 0 * * *" \
    td_sample_db \
    td_sample_table \
    load.yml

The `cron` parameter also accepts three special options: `@hourly`, `@daily` and `@monthly`.

By default, the schedule is set in the UTC time zone. You can set the schedule in a time zone using the `-t` or `--timezone` option. The `--timezone` option only supports extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected results.

List the Schedules

You can see the list of scheduled entries by running the command td connector:list.

$ td connector:list
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+
| Name         | Cron       | Timezone | Delay | Database     | Table           | Config                     |
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+
| daily_import | 10 0 * * * | UTC      | 0     | td_sample_db | td_sample_table | {"in"=>{"type"=>"amp", ... |
+--------------+------------+----------+-------+--------------+-----------------+----------------------------+

Show the Settings and Schedule History

td connector:show shows the execution settings of a schedule entry.

% td connector:show daily_import
Name     : daily_import
Cron     : 10 0 * * *
Timezone : UTC
Delay    : 0
Database : td_sample_db
Table    : td_sample_table
Config
---
in:
  type: amplitude
  api_key: "<YOUR_API_KEY>"
  secret_key: "<YOUR_SECRET_KEY>"
  start_time: "20160901T03" # UTC Timezone. The format is yyyymmddThh. "T" is a static character.
out:
  mode: append

td connector:history shows the execution history of a schedule entry. To investigate the results of each individual run, use td job <jobid>.

% td connector:history daily_import
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-18 00:10:05 +0000 | 160      |
| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-17 00:10:07 +0000 | 161      |
| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-16 00:10:03 +0000 | 152      |
| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-15 00:10:04 +0000 | 163      |
| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-14 00:10:04 +0000 | 164      |
| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-13 00:10:04 +0000 | 155      |
| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-12 00:10:05 +0000 | 156      |
| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2016-09-11 00:10:04 +0000 | 157      |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
8 rows in set

Delete the Schedule

td connector:delete removes the schedule.

$ td connector:delete daily_import

Configuration

See the following table for more details on available in modes.

Option name

Description

Type

Required?

Default value

api_key

API key

string

yes

N/A

secret_key

API Secret key

string

yes

N/A

start_time

First hour included in data series, formatted YYYYMMDDTHH (e.g. ‘20150201T05’). This is UTC timezone.

string

yes

N/A

end_time

Last hour included in data series, formatted YYYYMMDDTHH (e.g. ‘20150203T20’). This is UTC timezone.

string

no

processed time

incremental

true for “mode: append”, false for “mode: replace” (See below).

bool

no

true


Appendix

Modes for Out Plugin

You can specify the file import mode in the out section of seed.yml.

append (default)

This is the default mode, which appends records to the target table.

in:
  ...
out:
  mode: append

replace (In td 0.11.10 and later)

This mode replaces data in the target table. Any manual schema changes made to the target table remains intact with this mode.

in:
  ...
out:
  mode: replace
  • No labels