# Stripe Import Integration

This Data Connector allows you to import Stripe objects into Treasure Data.

## Prerequisites

- Basic knowledge of Treasure Data
- Basic knowledge of Stripe
- (Optional) [Stripe Webhooks](https://stripe.com/docs/webhooks)


## Use TD Console

### Create a New Connection

Go to Integrations Hub > Catalog and search and select Stripe.

![](/assets/image-20191022-133322.f1708963c0ac8dd299ee959b769a54a39b15cf76356100d24b5893fbc65345b0.b009b112.png)

We support the OAuth Authentication Methods:

Select an existing OAuth connection for Stripe, or select the link under **OAuth connection** to create a new one.

![](/assets/image-20191022-133342.a875b907d8c2aa56d61c432ccee952f62378e7081eed99e525ab15cb1085af54.b009b112.png)

### Create a New OAuth Connection

Login to your Stripe account:

![](/assets/image-20191022-133350.82d72fafa3becdba0d1fdc9d46f64755c0abebc282647bcb9c427f97d21a0f66.b009b112.png)

And grant access to **Treasure Data** app:

![](/assets/image-20191022-133409.e0bf18be978ea7127fcff40cc5b34d52ba8fd94d288614dbdcd6f329984a9519.b009b112.png)

You will be redirected back to Integrations Hub. Repeat Create a new connection and choose your new OAuth connection.

![](/assets/image-20191022-133418.35874dd688f32173c32059f02fc3361691877c27ccd171236e3ebb1338ebd154.b009b112.png)

Internal Testing Purposes: Secret Key —deprecated, replaced by OAuth

Enter Live Secret Key from Stripe > Your Account > Account Settings > API Keys.

![](/assets/image-20191022-133427.774f5d51765bb957b76a6cab4a28facdff4a98609859a05cb29cabfab4844b06.b009b112.png)

### Create a New Transfer

After creating the above connection, you will be automatically taken to [My Connections](https://console.treasuredata.com/connections/list) tab. Look for the connection you created and select New Transfer. The incremental option is only supported for the event object.

![](/assets/image-20191205-052042.d421627101de63282c5920ea15cd9600ec0e204a770542af27a0df7040f5743c.b009b112.png)

Edit the details and select `Next`.

![](/assets/image-20191022-133440.1a99949fe468212897207154efde3446e64625b4288e7c39a0c7169c9d4befb2.b009b112.png)

Preview your data. If you wish to change anything, select on Advanced Settings or else select on Next.

![](/assets/image-20191022-133521.10a8b688daddbc643113e510c9f3f9d8189ee64c197f495294ef5c165fd5eaac.b009b112.png)

Select the database and table where you want to transfer the data, as per the following dialog:

![](/assets/image-20191022-133627.f4a5e23e14b8d06eca32a7cae13a648a5e9c9ba0aa8d4de9a45ef94a7a262064.b009b112.png)

Specify the schedule of the data transfer using the dialog below and select **Start Transfer**.

![](/assets/image-20191022-133636.27a41d1bc0e053eca5eb3ae38f5a502276876a78a780c9d760fef9487a78485e.b009b112.png)

You see the new data transfer in progress listed under the My Input Transfers tab and a corresponding job are listed in the Jobs section.

### Parallel Data Import

You can improve data import performance by using `parallel import`.

The plugin creates a time range that depends on `Start datetime` and `End datetime` and send parallel HTTP requests to Stripe API. If you don't edit the `End datetime` plugin tries to get the timestamp of the latest record stored in Stripe first and use it as `End datetime`.

In the following example, the plugin creates 4 time ranges and all time ranges are processed in parallel.


```
Start datetime: 2016-01-25T00:00:00
End datetime: 2017-05-25T00.00.00
Time splitting
  Period for each parallel input: 6
  unit: month

Time range 1
  gte: 2016-01-25T00:00:00
  lt: 2016-01-31T00:00:00 # start_datetime to first end of the month
Time range 2
  gte: 2016-01-31T00:00:00 # has 6 month period
  lt: 2016-07-31T00.00.00
Time range 3
  gte: 2016-07-31T00.00.00.000Z # has 6 month period
  lt: 2017-02-28T00.00.00
Time range 4
  gte: 2017-02-28T00.00.00
  lt: 2017-05-25T00.00.00 # to end_datetime
```

![](/assets/image-20191022-133657.4ffb2d73758eeac183b574feb33f56b47978d8c37188e36799538b8e46d7baae.b009b112.png)

## Use Command Line

### Install ‘td’ command v0.11.9 or later

You can install the newest [TD Toolbelt](https://toolbelt.treasuredata.com/).


```
$ td --version
0.15.0
```

### Create Configuration File

Prepare configuration file (for eg: `load.yml`) as follows, with your Stripe account access information.


```yaml
in:
  type: stripe
  client_id: xxxxxxxxxxxxx
  client_secret: xxxxxxxxxxxxx
  refresh_token: xxxxxxxxxxxxx
  target: event
  incremental: true
  start_datetime: 2017-01-01T07:36:22.000Z
  formula: gt
out:
  mode: replace
```

The steps for specifying the import of Stripe `Event` objects is as follows:

1. client_id and client_secret: your Stripe app credentials
2. refresh_token: Stripe OAuth2 refresh_token, you need to grant access to your Stripe app, using an Stripe user account
3. target: Stripe object you want to import. Supported values: account, application_fee, balance_history, charge, coupon, customer, dispute, file_upload, event, invoice_item, invoice, order, plan, product, refund, subscription, transfer
4. start_datetime: import data from this date, format is: yyyy-MM-ddThh:mm:ss.000Z
5. formula: Formula to evaluate start_datetime. Supported values: gt(greater than), gte(greater than or equal), lt(less than), lte(less than or equal).
6. incremental: should data import be continuous or once, default as true. Incremental is only supported when target is event.


For more details on available out modes, see the Appendix.

### Preview Data to Import (Optional)

You can preview data to be imported using the command td connector:preview.


```bash
td connector:preview load.yml
```

### Execute Load Job

Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data are stored.

It is recommended to specify the --time-column option, since Treasure Data’s storage is partitioned by time (see also [data partitioning](https://docs.treasuredata.com/smart/project-product-documentation/data-partitioning-in-treasure-data)). If the option is not given, the Data Connector will choose the first long or timestamp column as the partitioning time. The type of the column specified for --time-column must be either of long or timestamp type.

If your data doesn’t have a time column you may add it using the add_time filter option. For more information, see [add_time filter plugin](https://docs.treasuredata.com/smart/project-product-documentation/add_time-filter-function).


```bash
td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column updated_date
```

The preceding command assumes you have already created *database(td_sample_db)* and *table(td_sample_table)*. If the database or the table do not exist in TD this command will not succeed, so create the database and table [manually](https://docs.treasuredata.com/smart/project-product-documentation/data-management) or use `--auto-create-table` option with the `td connector:issue` command to auto create the database and table:


```bash
td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column updated_date --auto-create-table
```

You can assign the Time Format column to the "Partitioning Key" with the "--time-column" option.

### Scheduled Execution

You can schedule a periodic Data Connector execution for a periodic Stripe import. We manage our scheduler to ensure high availability. By using this feature, you no longer need a `cron` daemon on your local data center.

A new schedule can be created using the `td connector:create` command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required.


```bash
$ td connector:create 
    daily_stripe_import 
    "10 0 * * *" 
    td_sample_db 
    td_sample_table 
    load.yml
```

The `cron` parameter also accepts these three options: `@hourly`, `@daily` and `@monthly`

By default, the schedule is setup in the UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option only supports extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules.

## Setup Stripe Webhooks

Webhooks can be used to capture events that happen in a Stripe account, instead of through direct API requests.

Login to Stripe account, select Webhooks —> Settings —> Add endpoint

![](/assets/image-20191022-133955.0879b08a954ec38e5cfaf929de2e1892b021dda355faea5a5109159d084d7f45.b009b112.png)

In the URL text box, enter:
http://in.treasuredata.com/postback/v3/event/{td_database}/{td_table}?td_write_key={td_account_write_api_key}

Then select events that you want to ingest into the {td_database}.{td_table} specified in the preceding URL.

![](/assets/image-20191022-134004.b94faca136abe830b97bb137c603bea498f2b2c0927c4b268ba6b2a283b7d47c.b009b112.png)

You can setup additional endpoints to ingest different types of events into different {td_database}.{td_table}.