TD Spark Driver (td-spark)

With td-spark driver you can access data sets in Treasure Data using Apache Spark.

Untitled-3
This feature is in BETA stage, and the access is disabled by default. We're looking for customers who know Apache Spark well and are willing to try this feature and give feedback to our team. If you're interested, please contact product@treasure-data.com or our support team.
Untitled-3
Recommendations regarding use For fastest data access, and lowest data transfer costs, we recommend that you set-up your spark cluster in the AWS us-east region. Data Transfer costs may become quite high if using other AWS regions or processing environments. While we anticipate giving this feature to all our customers by default, it may come at an additional expense once fully released if we see data access patterns that create additional expenses than anticipated.

Table of Contents

Download

The driver is avaialable from here: td-spark-assembly-latest.jar. This driver is recommended for use with Spark 2.1.0 or higher.

Release Notes

0.3.2

  • Oct 31st, 2017
  • Show td-spark version.

0.3.1

  • Oct 5th, 2017
  • Fixes a bug when reading tables containing null values.

0.3

  • July 14, 2017
  • Support full time range format, e.g., td.df("(database).(table)", from = "2017-07-14 01:00:00 PDT", to="2017-07-15 02:00:00 PDT")
  • Optimized DataFrame reader by using memory-efficient UnsafeRow class
  • Improved the performance and memory consumption when reading a single table schema
  • Fixed a bug that showed null values when reading columns that have alias names

0.2

  • Jan 25, 2017
  • Upgraded to Spark 2.1.0
  • Rename guava library package inside the assembly jar
  • Upgrade to Airframe 0.9, wvlet-log 1.1

0.1

  • Nov 17, 2016
  • Private alpha release
  • Added basic funcationalities for using TD dataset inside Spark
    • Reading mpc1 files
    • listing databases and table schemas
    • Run Presto/Hive queries
    • Suppport for SparkSQL
    • PySpark support

Last modified: Oct 31 2017 20:18:49 UTC

If this article is incorrect or outdated, or omits critical information, let us know. For all other issues, access our support channels.