Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

To enable this option, check Import Large Dataset then specify "Temp dataset", "Temp table", "GCS bucket" and "GCS path prefix". And please make sure that "Temp dataset" must be created manually in advance.


How This Works

  • When running a query (query loading or table loading with incremental loading), the query result is exported to a temporary BigQuery table "temp.temp_table".

  • Then the temp table is exported to "gs://my-bucket/data-connector/result-[12 digists number].jsonl.gz" as gzipped JSON Lines files. The number of files depends on the size of result data.

  • With a table loading without incremental loading, all the data in the source table is directly exported to GCS.

  • After completion, the temp table and GCS objects are deleted.

...