This feature is in Beta. Contact your Customer Success Representative for more information.
Live Connect Zero-Copy for Google BigQuery allows customers to use the Treasure Data Customer Data Platform (CDP) to access their data on BigQuery without the need for Extract, Transform, Load (ETL) processes. This approach enhances security and reduces costs associated with data movement. It enables seamless integration and querying of data stored in BigQuery directly from Treasure Data environments.
Performing zero-copy from the BigQuery Datawarehouse leverages the Treasure Data BigQuery integration to perform a federated query. The data flow looks like this:
Federated queries enable data engineers to connect to external data warehouses like using their credentials to access and query data. This functionality facilitates quick data augmentation and improves the efficiency of the data enrichment process. By using federated queries, data engineers can dynamically query and join external data with internal data based on current needs, providing flexibility and reducing the need for complex ETL processes. This approach also ensures real-time data access, improved data governance and security, and simplified data management.
At a high level, the steps you take to implement zero-copy with BigQuery are:
- In the TD Integrations Hub, create an authentication using the BigQuery integration.
- Create a New Zero-Copy source, specifying your JSON keyfile.
- In the TD Data Workbench, create a New Query. Preface the table name you want to query with the name of the zero-copy source you just created. For example, if you had a Zero-Copy (Federated Query) config named bigquery_test which was configured like this:
Then you could specify a query of table1 in the test database like this:
select * from bigquery_test.test_schema.table1
The test_schema is the BigQuery schema name.
- After running your query, process the results to update or augment your existing audience profiles.
After Treasure Data finishes processing the query results, the query results are deleted in seven days. However, you can use the INSERT INTO option to store the query results in Treasure Data.
Zero-copy with BigQuery has the following limitations:
- Catalog name length cannot exceed 64 characters.
- Catalog name prefix must start with bigquery_.
- There cannot be more than 200 "Zero-Copy (Federated Query) Configs" on the same account.
- The federated query is only available on Trino.
- The input size limitation per query is 10GB. (Not available in Trino preview clusters.)
- Catalog reflection time: Max 5 minutes(It takes up to 5 minutes to reflect a newly created zero-copy catalog or a change in a catalog.)