# Microsoft Azure Event Grid Streaming Import Integration

This feature is in BETA version. For more information, contact your Customer Success Representative.

Azure Event Grid is a fully managed event service that enables you to easily manage events across many different Azure services and applications made for performance and scale for event driven applications.
You can easily configure this integration to stream any event published into Azure Event Grid topics in near real-time, and have them in TD CDP for further decision-making.

## Prerequisites

- Basic Knowledge of Treasure Data.
- Basic knowledge of Azure Event Grid.
- [TD API Write-only Key](https://docs.treasuredata.com/smart/project-product-documentation/getting-your-api-keys)


## Limitations

- Custom events must include *"data": {}* to be correctly processed.
- SID is a combination of ASCII characters, digits and special symbols, and has to be a minimum of length 10.
- There is limitation of 2 source creations per account per connector type.
- Duplicate events are allowed.
- A Database and table need to be pre-created (unless the user has a master account).
- All limitations and constraints found in [Azure Event Grid delivery and retry - Azure Event Grid](https://docs.microsoft.com/en-us/azure/event-grid/delivery-and-retry) apply.
- The expected target latency for data to be available in Plazma is between 5  and 10 minutes.
- TD users are required to use the Write-only API key to create an Azure Event Grid Streaming connection.
- There is a few seconds delay before the newly created source starts streaming data.


## Use the TD Console to Create Your Connection

### Create a New Connection

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

1. Open **TD Console**.
2. Navigate to **Integrations Hub** >  **Catalog**.
3. Search for and select Azure Event Grid Data Streaming.
4. Select **Create Authentication.**


![](/assets/azureevengrid.8bbbbdb8e6714f3a183e889017f1e88e6e7688c13803a9f654b5677430aea82c.2f330e64.png)

The following dialog opens.

![](/assets/screen-shot-2021-09-24-at-1.14.30-pm.674676223b7ab77bd93dcc8a8362760817b7f45f54585767d8b0dfb874bd7a87.2f330e64.png)

Edit the following parameters:

| Parameters |  | Description |
|  --- | --- | --- |
| Security Identifiers (SIDs) | Required | SIDs are rotatable access tokens set by the user both in Azure Event Grid webhooks settings and create authentication model. It's  a combination of ASCII characters, digits and special symbols, and has to be minimum of length 10. Provide a comma separated list of tokens. |
| TD API Key | Required | TD write-only API Key for access to the database.  See [Getting Your API Keys](https://docs.treasuredata.com/smart/project-product-documentation/getting-your-api-keys). |


Security Identifiers(SIDs) will be masked. We recommend storing SIDs securely in your password manager for future rotations.

1. Select **Continue**.
2. Enter a name for your connection.
3. Select **Done.**


### Transfer Your Data to Treasure Data

After the authentication connection step, you are automatically taken to the Authentications listing page.

1. Search for the connection you created**.**
2. Select **New Source**.
3. The Connection dialog in the **Create Source** model opens.
4. Type a name for your **Source**in the Data Transfer Name field**.**
5. Click **Next**.  The Data Settings dialog opens.
6. Edit the following parameters:


| Parameters | Required | Description |
|  --- | --- | --- |
| Datastore | Required | Plazma is available option. |
| Tags | Optional | Tags can be used to find this source. |
| Database | Required | Specify the database within Treasure Data for which you want to import data into. |
| Table | Required | Specify the table within the database where you would like the data placed. |


1. Select **Create**.


### Copy the Source Id

The source id (UUID v4) is issued upon a Source creation.

To prevent misuse, the **Source Id** should not be disclosed to any unauthorized persons.

1. After creating the Source, you are automatically taken to the Sources listing page.


Search for the source you created.

Click on "..."  in the same row and Click **Copy Unique ID.** This Unique ID is the Source ID required when registering endpoint in Azure Event Grid.

![](/assets/screen-shot-2021-09-27-at-7.24.05-am.c3e75fd8f085ee8c5984e5e6914fffbb891ad7883a3854184af269e821f4099a.2f330e64.png)

### Endpoint Registration in Azure Event Grid

1. To have events processed by Azure Event Grid streaming connector, an endpoint has to be registered in Azure Event Grid with a valid Source Id. In Azure Event Grid, **Create Event Subscription** for Event Grid Topic.
2. Enter the Azure Event Grid endpoint in **ENDPOINT DETAILS.** The Azure Event Grid ingestion endpoints for the US use:  https://aeg-in-streaming.treasuredata.com/v1/task/{source id} Hostnames for other regions will differ but follow the same pattern.  For example, for Japan you use:  https://aeg-in-streaming.treasuredata.co.jp/v1/task/{source id}
3. Specify Security Identifiers (SIDs) as a static Authorization header. You register an SID from the list of Security Identifiers (SIDs) specified when creating an Authentication to enable rotations. Specify the SID as *Authorization: BEARER $SID* header in **Delivery Properties** tabunder **PROPERTIES SENT ON EVENT DELIVERY** section. Refer to the Quickstart guide from Azure Event Grid for complete instructions [Quickstart Portal](https://docs.microsoft.com/en-us/azure/event-grid/custom-event-quickstart-portal).
4. Customers can optionally specify event filtering for Event Grid Subscription.  There are three options for filtering:


- Event types
- Subject begins with or ends with
- Advanced fields and operators
![](/assets/screen-shot-2021-10-02-at-3.47.49-pm.931bdd4b00c4325fd493a3e3d489ee663503aed280ea58f7f7be419d5ccbe471.2f330e64.png)      
Please refer Azure Event Grid document for complete instructions [Event Filtering](https://docs.microsoft.com/en-us/azure/event-grid/event-filtering).


## Data Ingestion

The connector can start Event data ingestion in TD as soon as the first event is triggered after successful Streaming data source creation.