Page tree
Skip to end of metadata
Go to start of metadata

Treasure Data provides the following approaches for Machine Learning.



TD Console

Web-based user interface that requires little to no programming experience. Particularly designed for Marketers.


Query-based flexible and scalable way for everyone who knows SQL. Particularly designed for Data Engineers. This method involves using TD Console, Hivemall and Digdag.


pytd provides user-friendly interfaces to Treasure Data’s REST APIs, Presto query engine, and Plazma primary storage. Particularly designed for Data Scientists and Machine Learning Engineers. pytd allows Python code to efficiently read and write a large volume of data from and to Treasure Data.

TD Console

The TD Console is optimized for your company's customer data management and is designed for marketers who might not be familiar with how to code machine learning algorithms.

The application has the following machine learning features:

  • Content Affinity Engine - enables you to enrich customer data from customer behavior on websites.

  • Predictive Customer Scoring - detects high potential customers for marketing campaign focus.

Content Affinity Engine

The content affinity engine associates interest words to customers using web site information, including page title and description. For more details, see the article on data enrichment.

Predictive Scoring

Predictive scoring enables you to get potential customer segments, who are likely to buy, churn, click, or convert in near future. You can build a predictive model to find potential customers/prospects who likely to converge by defining conversions on customer behavior data. Learn more about predicting customer behavior.


Treasure Data machine learning provides flexible and scalable capability based on Apache Hivemall and Treasure Data Workflows. By using the TD Console to define and run your SQL queries, you can build your prediction model on your own. You can rapidly evolve machine learning tasks because there is no need to move data to and from Treasure Data.

The following is an example query that uses Hivemall to begin to train a classifier with logistic regression and predicting labels. The following example is from the Hivemall tutorial:

SELECT  train_classifier(    features,    label,    '-loss_function logloss -optimizer SGD'  ) as (feature, weight)FROM  training;

After you convert your table into pairs of features and label, you can build a binary classifier. When the table classifier has liner coefficients for given features, you can build the code to predict unforeseen samples by computing a weighted sum of their features. Prediction for the feature vectors can be made using a join operation between each feature in your model.

Hivemall supports the following machine learning tasks:

For how to build a machine learning model with Hivemall, see this tutorial or this regression tutorial.

Managing the Machine Learning Pipeline using Treasure Data Workflow

After ingesting your data into Treasure Data, you can build a predictive model using Treasure Data queries, workflows, and Hivemall.

The typical machine learning pipeline for supervised learning is:

  1. data preparation

  2. building a model

  3. evaluating the model

  4. predicting unseen data with trained model

For more information, refer to the wiki on supervised learning.

You can use Treasure Data Workflows to manage your supervised learning process. By using Digdag Treasure Data operators within your TD Workflow, you can automate your machine learning from data preparation to prediction. Digdag Treasure Data operators include:

Digdag can run tasks in parallel, so you can simultaneously run independent tasks such as parameter tuning. Treasure Data Workflows enable you to make prediction tasks a periodic part of your product offerings. Having a stable way to run and evolve your machine learning processes in batches on an hourly or daily basis is a good way to evolve them and derive a better predictive model.

Example machine learning workflows with Hivemall are available at Treasure Boxes.


Treasure Data provides an easy way to access Treasure Data data from your local Python environment with pytd. Using this method requires familiarity with:

  • Python, including its data structures and algorithms

  • pandas, an open source library for data structures and data analysis tools for Python

  • Jupyter, an open source library that supports interactive data science and scientific computing across all programming languages

Pandas-TD brings these tools together to help you build machine learning models to analyze Treasure Data data.

Pandas-TD is a Python library for interactive data analysis that uses Treasure Data as a data lake of time-series events, pandas for local DataFrame operations, and Jupyter to manage your analysis sessions. Pandas-TD also supports interactive analysis with Jupyter notebook.

Using Pandas-TD, you can fetch aggregated data from Treasure Data and move it into pandas. After creating a pandas DataFrame, you can visualize your data, and build a model with your favorite Python machine learning libraries such as scikit-learn, XGBoost, LightGBM, and TensorFlow.

Learn how to connect Treasure Data and pytd.pandas_td. You can use pytd.pandas_td for exploratory data analysis (EDA) before building machine learning models with Hivemall. For example see the Jupyter notebook portion of this GitHubGist.

  • No labels