You can use td-pyspark to bridge the results of data manipulations in Databrick with your data in Treasure Data.
Databricks builds on top of Apache Spark providing an easy to use interface for accessing Spark. PySpark is a Python API for Spark. Treasure Data's td-pyspark is a Python library that provides a handy way to use PySpark and Treasure Data based on td-spark.
To follow the steps in this example, you must have the following times:
Treasure Data API key
td-spark feature enabled
Configuring your Databricks Environment
You create a cluster, install td-pyspark libraries and configure a notebook for your connection code.
Create a Cluster on Databricks
Select the Cluster icon.
Select Create Cluster.
Provide a cluster name, select version Spark 2.4.3 or later as the Databricks Runtime Version and select 3 as the Python Version.
Install the td-pyspark Libraries
Access the Treasure Data Apache Spark Driver Release Notes for additional information and the most current download or select the link below.
Select to download
- td-spark-assembly-latest_spark2.4.7.jar (Spark 2.4.7, Scala 2.11)
- td-spark-assembly-latest_spark3.0.1.jar (Spark 3.0.1, Scala 2.12)
When the download completes, you see the following:
Specify your TD API Key and Site
In the Spark configuration, you specify the Treasure Data API key and enter the environment variables.
An example of the format is as follows. You provide the actual values:
Restart Cluster and Begin Work in Databricks
Restart your Spark cluster. Create a notebook. Create a script similar to the following code:
TDSparkContext is an entry point to access td_pyspark's functionalities. As shown in the preceding code sample, to create TDSparkContext, pass your SparkSession (spark) to TDSparkContext:
You see a result similar to the following:
Your connection is working.
Interacting with Treasure Data from Databricks
In Databricks, you can run select and insert queries to Treasure Data or query back data from Treasure Data. You can also create and delete databases and tables.
In Databricks you can use the following commands:
Read Tables as DataFrames
To read a table, use
Change the Database Used in Treasure Data
To change the context database, use
.df() your table data is read as Spark's DataFrame. The usage of the DataFrame is the same with PySpark. See also PySpark DataFrame documentation.
Submit Presto Queries
If your Spark cluster is small, reading all of the data as in-memory DataFrame might be difficult. In this case, you can use Presto, a distributed SQL query engine, to reduce the amount of data processing with PySpark.
Create or Drop a Database
Upload DataFrames to Treasure Data
To save your local DataFrames as a table, you have two options:
Insert the records in the input DataFrame to the target table
Create or replace the target table with the content of the input DataFrame
Checking Databricks in Treasure Data
You can use td toolbelt to check your database from a command line. Alternatively, if you have TD Console, you can check your databases and queries. Read about Database and Table Management.