This feature is in Beta. Contact your Customer Success Representative for more information.
Live Connect Zero-Copy for Amazon Redshift allows customers to use the Treasure Data Customer Data Platform (CDP) to access their data on Redshift without the need for Extract, Transform, Load (ETL) processes. This approach enhances security and reduces costs associated with data movement. It enables seamless integration and querying of data stored in Redshift directly from Treasure Data environments.
Performing zero-copy from the Redshft Datawarehouse leverages the Treasure Data Redshift integration to perform a federated query. The data flow looks like this:
Federated queries enable data engineers to connect to external data warehouses like using their credentials to access and query data. This functionality facilitates quick data augmentation and improves the efficiency of the data enrichment process. By using federated queries, data engineers can dynamically query and join external data with internal data based on current needs, providing flexibility and reducing the need for complex ETL processes. This approach also ensures real-time data access, improved data governance and security, and simplified data management.
At a high level, the steps you take to implement zero-copy with Redshift are:
- In the TD Integrations Hub, create an authentication using the Redshift integration.
- Create a New Zero-Copy source, specifying your Redshift credentials and connection information.
- In the TD Data Workbench, create a New Query. Preface the table name you want to query with the name of the zero-copy source you just created. For example, if you had a Zero-Copy (Federated Query) config named snowflake_conn which was configured like this:
Then you could specify a query of table1 in the test database like this:
select * from redshift_test.test_schema.table1
The test_schema is the Redshift schema name.
- After running your query, process the results to update or augment your existing audience profiles.
After Treasure Data finishes processing the query results, the query results are deleted in seven days. However, you can use the INSERT INTO option to store the query results in Treasure Data.
Zero-copy with Redshift has the following limitations:
- Catalog name length cannot exceed 64 characters.
- Catalog name prefix must start with redshift_.
- There cannot be more than 200 "Zero-Copy (Federated Query) Configs" across all DWHs on the same account.
- The federated query is only available on Trino.
- The input size limitation per query is 10GB. (Not available in Trino preview clusters.)
- Catalog reflection time: Max 5 minutes(It takes up to 5 minutes to reflect a newly created zero-copy catalog or a change in a catalog.)