# Facebook Ads Insights Import Integration

The Data Connector for Facebook Ads Insights (previous known as Facebook Ads Reporting) enables the import of data from your Facebook Advertising campaign results.

# Prerequisites

- Basic knowledge of Treasure Data
- Basic knowledge of Facebook Marketing API, in particular [Ads Insights API](https://developers.facebook.com/docs/marketing-api/reporting).
- Admin or Advertiser is required as admin permission


## Use TD Console

### Create a new connection

Go to [Treasure Data Connections](https://console.treasuredata.com/app/connections/my-connections) and search and select Facebook Ads. The following dialog opens.

![](/assets/image-20191017-181035.1577cc83f5875887b50ad38a67bdf8a6d5d95d993a4b9e160cc6050c17b8fc52.7ecd5fd4.png)

We support two Authentication Methods:

- OAuth
- Access Token


#### Using OAuth

Select an existing OAuth connection for Facebook or select the link under OAuth connection to create a new one.

![](/assets/image-20191017-181104.d6e7e16dd7f44db42198d496e64bf99984d3766c9505f7705939367defbe8047.7ecd5fd4.png)

- Create a new OAuth connection


Log into your Facebook account in a popup window:

![](/assets/image-20191017-181215.84b82feaced3680bb19707fc0476cccb5f24b8dd22ba66375fadc90e070dd4e6.7ecd5fd4.png)

And grant access to the Treasure Data app.

You will be redirected back to [Treasure Data Connections](https://console.treasuredata.com/app/connections/data-sources). Repeat the first step (Create a new connection) and choose your new OAuth connection.

![](/assets/image-20191017-181244.a60ae0b7f0aec4926d5bb6e9027725877936e6031873e0ece3d5fffe7045d67e.7ecd5fd4.png)

### Using an Access Token

See Obtain Facebook Token under Section 3 to create an access token.

### Create a new transfer

After creating the connection, you are automatically taken to the [Authentications](https://console.treasuredata.com/app/connections/my-connections) tab. Look for the connection you created and select New Source.

The following dialog opens.

![](/assets/image-20191017-181852.d2ec4e6047223b7cccc3332d17d8f8a421150cf341c65ffcf559706b30bec298.7ecd5fd4.png)

Edit the following details:

- Ad Account ID
- Data Level
- Import Metadata for Related Objects
- Import all Insights or Insights Fields
- Breakdowns
- Filtering
- Action Attribution Windows
- Start and End Date or Incremental
- Reporting Period


Details on these parameters are as follows:

#### Ad Account ID

Add your Ad Account ID, which you can find [as described here](https://www.facebook.com/business/help/1492627900875762).

#### Data Level

The Ads object level. Available options are [Campaign](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign-group), [Ad Set](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign), [and Ad](https://developers.facebook.com/docs/marketing-api/reference/adgroup).

#### Import Metadata for Related Objects

By default, Import metadata for related objects is unchecked and the connector imports only fundamental metadata fields: created_time, effective_status, status.

By checking it, you can specify related objects or individual metadata fields to import.

#### Related Edges

Select All Metadata Fields to import all related edges. You can select which edges to import by adding each edge in Fetch Metadata.

![](/assets/image-20191017-181943.0f8662c25dfaa1f79353be528fb0081ae03f2f43c1784f5f48f843c97839d9b0.7ecd5fd4.png)

#### Metadata Fields

Metadata fields associated with the Ads objects. Selecting Individual Metadata Fields allows you to import only the fields you want. Select fields under Metadata Fields.

![](/assets/image-20191017-182014.d8c6197ead093910d4d366530fdddb591b4ce6f4d99ef6c89786fd5206b6d49b.7ecd5fd4.png)

You can also access Ads object’s edges by using edge{field1, field2…}.

#### Insights Fields

Insight fields provide data about the performance of your campaign.

By default, Import all insights fields are checked.

![](/assets/image-20191017-182145.5f85958eaeb7374abdbeeacce093f8279dfffbce1dd8511fce87603fe0cbdf23.7ecd5fd4.png)

You can uncheck Import all insights fields to select specific fields.

![](/assets/image-20191017-182233.abf14a3a7aea654749a46b8a4e89bc52d41bb3cdd36adc1bf7eab279f0553e0e.7ecd5fd4.png)

The [Insights](https://developers.facebook.com/docs/marketing-api/insights/parameters#fields) fields date_start and date_stop are added by default, if not explicitly set. Insight values might be empty depending on the state of Ads object. No errors are generated if the fields do not contain values.

#### Breakdowns

Breakdowns are used to capture Ads Insights responses that are grouped into different sets or cohorts. Supported breakdowns are:

- age
- gender
- country
- region
- hourly_stats_aggregated_by_advertiser_time_zone
![](/assets/image-20191017-182602.176baa834f7a5c988fb153840b1f707717ea1aa4bd2e2d5cf56833bb82c43cc9.7ecd5fd4.png)


#### Filtering

Filters on the reported Insights data. Format: Field —> Operator —> Value, the action_type supports IN operator with comma separate values, i.e. post_like, like

![](/assets/image-20191017-182636.70e3b86ff9bd63c4aebb6d29997f5c20aac293a1785898c2008fad41a77ae1d3.7ecd5fd4.png)

#### Action Attribution Windows

Determines the attribution window for the actions. Supported values are:

- 1d_view
- 7d_view
- 28d_view
- 1d_click
- 7d_click
- 28d_click


Defaults values are: ([“1d_view”,“28d_click”]) if not explicitly set.

![](/assets/image-20191017-182744.08b30b683726638e208b213f3e8165b5015c618e91133cf1158977685e764000.7ecd5fd4.png)

### Incremental Import and Start Date

Specify if you want **Incremental** import and **Start Date**.

By enabling incremental, you can define the duration (in days) of each incremental load. The default is 1 day. If **Start Date** is not specified, then the default is set to past 30 days.

The data that is fetched includes data that is available up until midnight of the previous night.

### Reporting Period

After you pick a reporting period by using **Start Date** and **End Date**, you may choose to get results for the whole period or get results for smaller time slices. If you select **All Days**, then you get one result set for the whole period. If you specify **monthly**, then you get one result set for each calendar month in the given period.

![](/assets/image-20191017-182818.b3f389943b38e920575a9742ccabc8647a09c5e7fc4d53df09cba266e5c55653.7ecd5fd4.png)![](/assets/image-20191017-182855.3ce754b59a79079685b577b72b2ea1e57e42d4b1585e06692418734f362cfdcf.7ecd5fd4.png)

Or you can get one result set for each N-day period specified by selecting **In Days**. You can specify the number of days from 1 to 90.

![](/assets/image-20191017-182922.3ce754b59a79079685b577b72b2ea1e57e42d4b1585e06692418734f362cfdcf.7ecd5fd4.png)

After you complete specifying parameters, select **Next**. You then see a Preview of your data similar to the dialog below. Select **Next**. Select [Preview while Importing Data](https://docs.treasuredata.com/smart/project-product-documentation/about-data-preview) if you cannot move beyond Preview.

![](/assets/image-20191017-183241.0cc37aaa8b7ebcd6de0c81a800b22ae833fd9e18777c6196fc6a306ec8c4ea45.7ecd5fd4.png)

Select the database and table where you want to transfer the data, as shown in the following dialog:

![](/assets/image-20191017-183551.7daa417c4552d8d9afeb062c627880a591e65eb2edac38f029a5fe7f8f85301d.7ecd5fd4.png)

Specify the schedule of the data transfer using the dialog below and select **Start Transfer**.

![](/assets/image-20191017-183622.02f17beca8df86817524a2635baefb2e2afd3e513414ef6af7cb557ca028dbd9.7ecd5fd4.png)

You see the new data transfer in progress listed under the My Input Transfers tab and a corresponding job is listed in the Jobs section.

Now, you are ready to start analyzing your data!

## Use Command Line

### Install TD Toolbelt

Install the newest TD Toolbelt.

### Obtain Facebook Token

[Facebook provides 3 types of tokens](https://developers.facebook.com/docs/facebook-login/access-tokens). We recommend using the never-expiring Page Access Token. You can also use User Token but it expires after 2 months even if it is extended. It is not batch-friendly.

For steps that describe how to get the never-expiring Page Access Token, see [Facebook Long-Lived Access Tokens](https://developers.facebook.com/docs/facebook-login/guides/access-tokens/get-long-lived/).

### Prepare Configuration and Preview Data

We don't need to run td connector:guess for the facebook_ads_reporting connector.

Prepare config.yml as follows:


```yaml
in:
  type: "facebook_ads_reporting"
  ad_account_id: '[your ad account id]'
  access_token: "[your Facebook access token]"
  data_level: ad
  fields:
    - clicks
    - impressions
  metadata_import: all
  metadata_edge:
    - adrules_governed
    - copies
    - keywordstats
    - leads
    - targetingsentencelines
  filtering:
    - {field : "clicks", operator: "GREATER_THAN", value: 10}
    - {field : "impressions", operator: "GREATER_THAN", value: 200}
out:
  mode: append
```

You can see a preview of the data using the `preview` command.


```bash
td connector:preview config.yml
```

### Configuration

See the following table for more details on available modes.

| **Option name** | **Description** | **Type** | **Required?** | **Default value** |
|  --- | --- | --- | --- | --- |
| ad_account_id | Ad account ID of access_token | string | yes | — |
| access_token | Access Token. We recommend using the non-expiring token (see the next section) | string | yes | — |
| app_secret | If set, appsecret_proof parameter is used to access API | string | optional | — |
| metadata_import | Import related edges or individual metadata fields? Available options are all and individual | string | optional | individual |
| metadata_edge | Related edges to import. Only effective when metadata_import is all, see related edges. | array | optional | — |
| metadata_fields | What metadata fields you want to obtain for a given object. [Review available fields.](https://developers.facebook.com/docs/marketing-api/reference/v2.9) | array | optional | created_time, status, effective_status |
| fields | What fields you want to obtain for insights. [Review available fields.](https://developers.facebook.com/docs/marketing-api/reference/v2.9) | array | optional | — |
| breakdowns | See breakdowns for details. | string | optional | — |
| filtering | See filter for details. | array | optional | — |
| action_attribution_windows | Determines the attribution window for the actions. For example, 28d_click Means the API returns all actions that happened 28 days after someone clicked on the ad. | array | optional | — |
| retry_limit | Trying maximum retry times | integer | optional | 5 |
| retry_initial_wait_sec | Wait seconds for exponential backoff initial value | integer | optional | 10 |
| last_fetched_at | Only imports data since this date, it is 30 days from today if not specified, ie. (today - 30 days). Recommended formats: %m/%d/%Y or %Y-%m-%dT%H:%M:%S.%L%Z | string | optional | — |
| incremental | true for generate “config_diff” with embulk run -c config.diff | bool | optional | true |
| duration | # of days from start date to pull in insights data. See Appendix B for more details. | integer | optional | 1 |
| time_increment | If it is an integer, it is the number of days from 1 to 90. After you pick a reporting period by using last_fetched_at and last_fetched_until, you might choose to have the results for the whole period, or have results for smaller time slices. If “all_days” is used, it means one result set for the whole period. If “monthly” is used, you get one result set for each calendar month in the given period. Or you can have one result set for each N-day period specified by this parameter. See Appendix B for more details. | string | optional | 1 |
| last_fetched_until | Only imports data until this date, today if not specified. Recommended formats: see last_fetched_at. See Appendix C for more details. | string | optional | — |
| add_time_column | If true, “time” column is added automatically, based on date_stop value of insights, otherwise, “time” column is added by upload_time | bool | optional | true |
| data_level | Obtain data level. Facebook insights level: Campaign –> Adset –> Ad. Available options are “campaign”, “adset”, “ad”, or “ad_creative”. | string | optional | “ad” |
| api_version | Use Facebook API version. Default is v3.2. If specified version is older than v3.1 (deprecated), plugin uses “v3.2” | string | optional | “v3.2” |
| limit_per_page | Number of records per page, reduce this value when you encounter error message “Please reduce the amount of data you’re asking for, then retry your request” | integer | optional | 100000 |


### Supported Related Edges

#### Campaign

- ad_studies
- adrules_governed
- ads
- adsets
- copies


#### AdSet

- activities
- ad_studies
- adrules_governed
- ads
- copies
- targetingsentencelines


#### Ad

- adrules_governed
- copies
- keywordstats
- leads
- targetingsentencelines


### Duration and time_increment

#### When to use duration?

- The **duration** has effect only when **incremental = true**
- The **duration** is used to compute time range for your request, based on **last_fetched_at**, for example, if **last_fetched_at = 02/01/2017** and **duration = 2**, you’re requesting for a time range of: 02/01/2017 → 02/02/2017


Directly translated to Facebook request parameters:


```
.time_range({"since":"2017-02-01","until":"2017-02-02"})
```

- There is no breakdown behavior when you use this parameter. It returns insight values for the whole time range.
- **Note**: When no **last_fetched_at** is specified, it is 30 days from today, ie. (today – 30 days)


#### When to use time_increment?

- **time_increment** is used when you want to break down insights data by a number of days. The results are easiest to recognize without incremental mode and duration.


Example:


```yaml
incremental: false
last_fetched_at: 03/01/2017
last_fetched_until: 03/31/2017
time_increment: 1
```

Using the preceding configuration, you can import the entire month of March 2017 in one run, and still have it break down daily.

#### duration and time_increment differences

- **duration** is used for **incremental** mode
- **time_increment** is an official Facebook parameter
- **Important note**: If both **duration** and **time_increment** are available (which is default), **duration** will have higher priority than **time_increment** when its value is less than **time_increment**. With that being said, you can’t have breakdown data per 2 days (**time_increment = 2**) when **duration = 1**, result will be 1-day data. In the other hand, you can perfectly break down data daily (**time_increment = 1**) for a 2-day duration (**duration = 2**).


#### When to use last_fetched_until?

- **last_fetched_until** is optional, and its only usage is to limit imported data to an end date.


Example: you have a job to import March 2017, and it’s scheduled to run repeatedly to get daily data. Therefore, you could use the following configuration to achieve it:


```yaml
incremental: true
duration: 1
last_fetched_at: 03/01/2017
last_fetched_until: 03/31/2017
```

**Note**: As you can see, you can simulate **time_increment** effect by using **duration** and run the job repeatedly, but it will cost you more requests (Request Rate Limit will occur faster).

- If your scheduled job (eventually) exceeds configured **last_fetched_until**, an empty result is returned.


## Data Connector schema change in the July 10 2018 release

Due to recent changes of Facebook API, Data Connector has updated.

**Before the change:** Some fields were added but the output schema was missing some columns because the value was empty.

**After the change:** Some new columns are added but might not contain values. The output schema will be combined from the input configuration and mandatory fields.

The mandatory fields and implicitly added columns are:

- Metadata fields
  - created_time
  - effective_status
  - status
- Insights Fields
  - date_start
  - date_stop


Invalid `**fields**` that might have been skipped in previous updates will now result in errors after update. See the `Fields` section of following links for complete references for Insights Fields and Metadata fields:

- [Metadata fields for Campaign level](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign-group#Reading)
- [Metadata fields for Ad Set level](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign#Reading)
- [Metadata field for Ad level](https://developers.facebook.com/docs/marketing-api/reference/adgroup#Reading)
- [Insights fields](https://developers.facebook.com/docs/marketing-api/reference/ads-insights/)