This feature is in beta. Contact your Customer Success representative for more information.


Azure Event Grid is a fully managed event service that enables you to easily manage events across many different Azure services and applications made for performance and scale for event driven applications.
You can easily configure this integration to stream any event published into Azure Event Grid topics in near real-time, and have them in TD CDP for further decision-making.

Learn more about:

Prerequisites

Limitations

  • Custom events must include "data": {} to be correctly processed.

  • SID is a combination of ASCII characters, digits and special symbols, and has to be a minimum of length 10.

  • There is limitation of 2 source creations per account per connector type. 

  • Duplicate events are allowed.

  • A Database and table need to be pre-created (unless the user has a master account).

  • All limitations and constraints found in Azure Event Grid delivery and retry - Azure Event Grid apply.

  • The expected target latency for data to be available in Plazma is between 5  and 10 minutes.
  • TD users are required to use the Write-only API key to create an Azure Event Grid Streaming connection.
  • There is a few seconds delay before the newly created source starts streaming data.

Use the TD Console to Create Your Connection

Create a New Connection

In Treasure Data, you must create and configure the data connection before running your query. As part of the data connection, you provide authentication to access the integration.

1. Open TD Console.
2. Navigate to Integrations Hub  Catalog.
3. Search for and select Azure Event Grid Data Streaming.

4. Select Create Authentication.

          The following dialog opens.

5. Edit the following parameters:

Parameters 


Description

Security Identifiers (SIDs)RequiredSIDs are rotatable access tokens set by the user both in Azure Event Grid webhooks settings and create authentication model. It's  a combination of ASCII characters, digits and special symbols, and has to be minimum of length 10. Provide a comma separated list of tokens.
TD API KeyRequired

TD write-only API Key for access to the database.

See Getting Your API Keys.

Security Identifiers(SIDs) will be masked. We recommend storing SIDs securely in your password manager for future rotations.

6. Select Continue.
7. Enter a name for your connection.
8. Select Done.

Transfer Your Data to Treasure Data

After the authentication connection step, you are automatically taken to the Authentications listing page.

1. Search for the connection you created.
2. Select New Source.
3. The Connection dialog in the Create Source model opens.
4. Type a name for your Source in the Data Transfer Name field.

5. Click NextThe Data Settings dialog opens.
6.  Edit the following parameters:

Parameters 


Description

DatastoreRequiredPlazma is available option.
TagsOptionalTags can be used to find this source.
DatabaseRequiredSpecify the database within Treasure Data for which you want to import data into.
TableRequiredSpecify the table within the database where you would like the data placed.
7. Select Create.

Copy the Source Id

The source id (UUID v4) is issued upon a Source creation.

To prevent misuse, the Source Id should not be disclosed to any unauthorized persons.

1. After creating the Source, you are automatically taken to the Sources listing page.
2. Search for the source you created.
3. Click on "..."  in the same row and Click Copy Unique ID. This Unique ID is the Source ID required when registering endpoint in Azure Event Grid.

Endpoint Registration in Azure Event Grid

1. To have events processed by Azure Event Grid streaming connector, an endpoint has to be registered in Azure Event Grid with a valid Source Id. In Azure Event Grid, Create Event Subscription for Event Grid Topic.

2. Enter the Azure Event Grid endpoint in ENDPOINT DETAILS. The Azure Event Grid ingestion endpoints for the US use:  https://aeg-in-streaming.treasuredata.com/v1/task/{source id}

Hostnames for other regions will differ but follow the same pattern.  For example, for Japan you use:  https://aeg-in-streaming.treasuredata.co.jp/v1/task/{source id}
3.  Specify Security Identifiers (SIDs) as a static Authorization header. You register an SID from the list of Security Identifiers (SIDs) specified when creating an Authentication to enable rotations.

Specify the SID as Authorization: BEARER $SID header in Delivery Properties tab under PROPERTIES SENT ON EVENT DELIVERY section. Refer to the Quickstart guide from Azure Event Grid for complete instructions Quickstart Portal.
4.    Customers can optionally specify event filtering for Event Grid Subscription.  There are three options for filtering:
    • Event types
    • Subject begins with or ends with
    • Advanced fields and operators
           
      Please refer Azure Event Grid document for complete instructions Event Filtering.

Data Ingestion

The connector can start Event data ingestion in TD as soon as the first event is triggered after successful Streaming data source creation.




  • No labels