Google Bigquery Export Integration V2

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

The Google BigQuery Connector V2 is designed to streamline the process of uploading large volumes of data to Google BigQuery. It offers the following key capabilities:

• Efficient packaging of datasets into Parquet files to handle large volumes of data uploaded to Big Query
• Optimized data uploads using BigQuery load jobs
• Flexible data sync operations, with added support for Truncate sync mode.

Prerequisites

• Basic knowledge of Treasure Data, including the TD Toolbelt.
• A Google Cloud Platform account

Requirements and Limitations

• Nested or repeated data types like ARRAY are not supported as destination columns.

Supported

This connector supports the "append, replace, replace backup, truncate" modes.

Obtaining your Google Cloud Platform Credentials

To use this feature, you need:

• Project ID
• JSON Credential

Obtaining the JSON Credential

The integration with Google BigQuery is based on server-to-server API authentication.

1. Navigate to your Google Developer Console.
2. Select APIs & auth > Credentials.
3. Select Service account.

4. Select the JSON-based key type, which is recommended by Google. The key is downloaded automatically by the browser.

Obtaining the Project ID

1. Navigate to your Google Developer Console.
2. Select Home.
3. Locate your Project ID.

Create a Dataset and Table on BigQuery

Create your Dataset and Table from your BigQuery console.

Use from TD Console

1. Navigate to TD Console.
2. Navigate to Integrations Hub > Catalog.
3. Select Google Big Query V2.

4. Complete all the information as follows:

Configure a Query Result for Export

The TD Console supports multiple ways to export data. Follow these steps to export data from the Data Workbench.

1. Navigate to Data Workbench > Queries.
2. Select New Query, and define your query.
3. Select Export Results to configure the data exporting.
4. Select an existing Snapchat CAPI authentication or create a new one described previously
5. Select Done.

Connector Configuration Parameters

Query Result Data Specs

BigQuery Table Schema Reconciliation and Validation

The connector implements a schema reconciliation or consolidation process that harmonizes schemas from multiple sources to determine the final table structure or schema. This process ensures data integrity while supporting schema evolution scenarios based on configured sync modes and user preferences.

Schema Sources and Precedence Hierarchy

The connector determines the final table schema by evaluating three potential schema sources in a hierarchical order. This hierarchy is crucial for understanding how your data schema will be materialized in the destination BigQuery table.

1. Destination Table Schema (Highest Precedence)

If the destination table exists, its schema serves as the primary authority for data loading operations.

When it applies:

• During APPEND operations to existing tables
• During TRUNCATE operations

Practical Example: Numeric Type Handling

• Query Resut Data

Destination Table Schema

• user_id: INT64
• transaction_amount: NUMERIC(38,9)
• transaction_date: DATE

Result: Even if query result data comes as FLOAT64 for transaction_amount, it will be converted to NUMERIC as per destination schema.

2. User-Defined JSON Schema (Second Precedence)

When provided through the JSON Schema File configuration, this schema overrides type inference from query results and provides explicit column definitions.

When it applies:

• During table creation (new tables)
• When adding new columns to existing tables
• For explicit type casting and column property definition​

Practical Example: Timestamp and Date Conversions

• Source Data

• User-Defined Schema

• event_time: TIMESTAMP

• registration_date: DATE

• Effects

event_time:

• Source data will be parsed as TIMESTAMP
• Maintains microsecond precision
• Stores in UTC

3. Query Result Schema (Lowest Precedence)

The schema derived from your source data structure. This serves as the baseline schema when no other schemas are specified.

When it applies:

• When creating new tables without a JSON schema definition
• When adding new columns without explicit type definitions
• As a source for type inference when no overrides exist

Practical Example: Integer to String Conversion

• Query Result Data

• Query Result Schema

• product_id: STRING

• status_code: Int

• Possible Target Conversions

1. status_code as STRING:

• INT64 → STRING (Compatible conversion)
• Result: “200”, “404”

2. status_code as INT64:

• Maintains original type
• Result: 200, 404

Schema Handling Scanario Matrix

Find below a structured reference table showing exactly how schemas are handled in each scenario, organized by sync mode, table state, and configuration settings.

Data Type Mapping

The connector implements a comprehensive type system mapping between source data types and BigQuery native types, with three categories of conversions:

1. Default Mappings (Lossless Conversions)

2. Compatible Mappings (Type Coercion)

3. Potentially Lossy Mappings (Requires Validation)

(Optional) Schedule Query Export Jobs

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

Treasure Data's scheduler feature supports periodic query execution to achieve high availability.

When two specifications provide conflicting schedule specifications, the specification requesting to execute more often is followed while the other schedule specification is ignored.

For example, if the cron schedule is '0 0 1 * 1', then the 'day of month' specification and 'day of week' are discordant because the former specification requires it to run every first day of each month at midnight (00:00), while the latter specification requires it to run every Monday at midnight (00:00). The latter specification is followed.

Scheduling your Job Using TD Console

1. Navigate to Data Workbench > Queries

2. Create a new query or select an existing query.

3. Next to Schedule, select None.

4. In the drop-down, select one of the following schedule options:

   Drop-down Value	Description
   Custom cron...	Review Custom cron... details.
   @daily (midnight)	Run once a day at midnight (00:00 am) in the specified time zone.
   @hourly (:00)	Run every hour at 00 minutes.
   None	No schedule.

Custom cron... Details

 *    *    *    *    *
 -    -    -    -    -
 |    |    |    |    |
 |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 |    |    |    +---------- month (1 - 12)
 |    |    +--------------- day of month (1 - 31)
 |    +-------------------- hour (0 - 23)
 +------------------------- min (0 - 59)

The following named entries can be used:

• Day of Week: sun, mon, tue, wed, thu, fri, sat.
• Month: jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec.

A single space is required between each field. The values for each field can be composed of:

1. (Optional) You can delay the start time of a query by enabling the Delay execution.

Execute the Query

Save the query with a name and run, or just run the query. Upon successful completion of the query, the query result is automatically exported to the specified destination.

(Optional) You can delay the start time of a query by enabling the Delay execution.

Activate a Segment in Audience Studio

You can also send segment data to the target platform by creating an activation in the Audience Studio.

1. Navigate to Audience Studio.
2. Select a parent segment.
3. Open the target segment, right-mouse click, and then select Create Activation.
4. In the Details panel, enter an Activation name and configure the activation according to the previous section on Configuration Parameters.
5. Customize the activation output in the Output Mapping panel.

• Attribute Columns
  • Select Export All Columns to export all columns without making any changes.
  • Select + Add Columns to add specific columns for the export. The Output Column Name pre-populates with the same Source column name. You can update the Output Column Name. Continue to select + Add Columnsto add new columns for your activation output.
• String Builder
  • + Add string to create strings for export. Select from the following values:
    • String: Choose any value; use text to create a custom value.
    • Timestamp: The date and time of the export.
    • Segment Id: The segment ID number.
    • Segment Name: The segment name.
    • Audience Id: The parent segment number.

6. Set a Schedule.

• Select the values to define your schedule and optionally include email notifications.

7. Select Create.

(Optional) Export Integration Using the CLI

You can also use CLI (Toolbelt) to export results to BigQuery.

You need to specify the information for export to your Snapchat server using the *--result option of*the td query command. For more information about the *td query*command, refer to this article.

The format of the option is JSON, and the general structure is as follows.

APPEND mode:

type: 'bigquery_v2'
json_keyfile: |
  {
    "type": "service_account",
    "private_key_id": "xxx",
    "private_key": "-----BEGIN PRIVATE KEY-----xxx-----END PRIVATE KEY-----\n",
    "client_email": "account@xxx.iam.gserviceaccount.com",
    "client_id": "xxx",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://accounts.google.com/o/oauth2/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40xxx.iam.gserviceaccount.com"
  }
mode: APPEND
project: gcp project id
dataset: bigquery dataset
table: bigquery table
location: gcp location
auto_create_table: true
add_missing_columns: true
schema_file: |
  [
    {"name": "c1", "type": "STRING", "mode": "NULLABLE"},
    {"name": "c2", "type": "INTEGER", "mode": "REQUIRED"}
  ]
skip_invalid_records: true

REPLACE mode:

  type: 'bigquery_v2'
  json_keyfile: |
    {
      "type": "service_account",
      "private_key_id": "xxx",
      "private_key": "-----BEGIN PRIVATE KEY-----xxx-----END PRIVATE KEY-----\n",
      "client_email": "account@xxx.iam.gserviceaccount.com",
      "client_id": "xxx",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://accounts.google.com/o/oauth2/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40xxx.iam.gserviceaccount.com"
    }
  mode: REPLACE
  project: gcp project id
  dataset: bigquery dataset
  table: bigquery table
  location: gcp location
  auto_create_table: true
  add_missing_columns: true
  schema_file: |
    [
      {"name": "c1", "type": "STRING", "mode": "NULLABLE"},
      {"name": "c2", "type": "INTEGER", "mode": "REQUIRED"}
    ]
  skip_invalid_records: true

REPLACE_BACKUP mode:

  type: 'bigquery_v2'
  json_keyfile: |
    {
      "type": "service_account",
      "private_key_id": "xxx",
      "private_key": "-----BEGIN PRIVATE KEY-----xxx-----END PRIVATE KEY-----\n",
      "client_email": "account@xxx.iam.gserviceaccount.com",
      "client_id": "xxx",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://accounts.google.com/o/oauth2/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40xxx.iam.gserviceaccount.com"
    }
  mode: REPLACE_BACKUP
  backup_mode: TABLE_RENAME
  project: gcp project id
  dataset: bigquery dataset
  table: bigquery table
  location: gcp location
  auto_create_table: true
  add_missing_columns: true
  schema_file: |
    [
      {"name": "c1", "type": "STRING", "mode": "NULLABLE"},
      {"name": "c2", "type": "INTEGER", "mode": "REQUIRED"}
    ]
  skip_invalid_records: true

TRUNCATE mode:

  type: 'bigquery_v2'
  json_keyfile: |
    {
      "type": "service_account",
      "private_key_id": "xxx",
      "private_key": "-----BEGIN PRIVATE KEY-----xxx-----END PRIVATE KEY-----\n",
      "client_email": "account@xxx.iam.gserviceaccount.com",
      "client_id": "xxx",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://accounts.google.com/o/oauth2/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40xxx.iam.gserviceaccount.com"
    }
  mode: TRUNCATE
  project: gcp project id
  dataset: bigquery dataset
  table: bigquery table
  location: gcp location
  auto_create_table: true
  add_missing_columns: true
  schema_file: |
    [
      {"name": "c1", "type": "STRING", "mode": "NULLABLE"},
      {"name": "c2", "type": "INTEGER", "mode": "REQUIRED"}
    ]
  skip_invalid_records: true

Parameters

Example for Usage

APPEND mode

$ td query --result '{"type": "bigquery_v2", "td_authentication_id": 123456, "mode": "APPEND", "project": "gcp_project_id", "dataset": "bg_dataset", "table": "bg_table", "location": "US", "auto_create_table": true, "add_missing_columns": true, "schema_file": "[{\"name\": \"c1\", \"type\": \"INTEGER\"}, {\"name\": \"c2\", \"type\": \"STRING\"}]", "skip_invalid_records":true}' -d sample_datasets "select ........ from ........" -T presto

REPLACE mode

$ td query --result '{"type": "bigquery_v2", "td_authentication_id": 123456, "mode": "REPLACE", "project": "gcp_project_id", "dataset": "bg_dataset", "table": "bg_table", "location": "US", "auto_create_table": true, "add_missing_columns": true, "schema_file": "[{\"name\": \"c1\", \"type\": \"INTEGER\"}, {\"name\": \"c2\", \"type\": \"STRING\"}]", "skip_invalid_records":true}' -d sample_datasets "select ........ from ........" -T presto

REPLACE_BACKUP mode

$ td query --result '{"type": "bigquery_v2", "td_authentication_id": 123456, "mode": "REPLACE_BACKUP", "backup_mode": "TABLE_RENAME", "project": "gcp_project_id", "dataset": "bg_dataset", "table": "bg_table", "location": "US", "auto_create_table": true, "add_missing_columns": true, "schema_file": "[{\"name\": \"c1\", \"type\": \"INTEGER\"}, {\"name\": \"c2\", \"type\": \"STRING\"}]", "skip_invalid_records":true}' -d sample_datasets "select ........ from ........" -T presto

TRUNCATE mode

$ td query --result '{"type": "bigquery_v2", "td_authentication_id": 123456, "mode": "TRUNCATE", "project": "gcp_project_id", "dataset": "bg_dataset", "table": "bg_table", "location": "US", "auto_create_table": true, "add_missing_columns": true, "schema_file": "[{\"name\": \"c1\", \"type\": \"INTEGER\"}, {\"name\": \"c2\", \"type\": \"STRING\"}]", "skip_invalid_records":true}' -d sample_datasets "select ........ from ........" -T presto

Related articles

Other Configurations

• The Result Export can be scheduled to periodically upload data to a target destination.
• All import and export integrations can be added to a TD Workflow. The td data operator can export a query result to a specified integration. For more information, see Reference for Treasure Data Operators.