This integration allows you to export TD job results into your existing MongoDB instance. For sample workflows on this, please view Treasure Boxes.

Prerequisites

  • Basic knowledge of Treasure Data, including the TD Toolbelt.

  • A MongoDB instance.

  • Treasure Data must have the proper privileges.

Reference Architecture

A front-end application collects data into Treasure Data via the Treasure Agent. Treasure Data periodically runs jobs on the data, then writes the job results to your MongoDB collections.

Example 1: Ranking: What are the “Top N of X?”

Every social/mobile application calculates the “top N of X” (e.g., top 5 movies watched today). Treasure Data already handles the raw data warehousing; the “write-to-mongodb” feature also enables Treasure Data to find the “top N” data.

Example 2: Dashboard Application

If you are a data scientist, you need to keep track of a range of metrics every hour/day/month and make them accessible via visualizations. Using this “write-to-mongodb” feature, you can streamline the process and focus on your queries and visualizations of the query results.

You can limit access to your MongoDB instance by using a list of static IPs. Contact support@treasuredata.com if you require a static IP.


Basic Usage

  1. Navigate to the Treasure Data Console > Query editor, and enter your query.

  2. Select Output Results.


You have 2 options when creating a result output connector:

    • Use an Existing Connection, or
    • Create a New Connection.

Use an Existing Connection

Type the name of the connection in the prompt and select it.

Create a New Connection


  • Host: The hostname or IP address of the remote server. (You can add more than one IP address, depending on your MongoDB setup.)

  • Username: Username to connect to the remote database.

  • Password: Password to connect to the remote database.

  • Use TLS?:  Check this box to connect using TLS (SSL).

  • Bypass certificate validations?: Check this box to bypass all certificate validations.

  • Database name: The name of the database to which you are transferring data (e.g., your_database_name).

  • Table Name: The name of the collection to which you are transferring data.

  • Mode:

    • Append - Add to the existing records in the database. This mode is atomic.

    • Replace - Replace the existing records with the query results. This mode is atomic.

    • Truncate - Truncate the existing records. This mode is atomic.


When you execute your query, Treasure Data query's results are automatically imported into your MongoDB instance.

Exporting to MongoDB from a Workflow

In Workflow definitions, you can export the result of a query to MongoDB by declaring your connection using the result_connection and result_settings options. You can lean more about modes in detail.

#Example Worflow
_export:
result_connector_name: mongodb_connector
target_database_name: mongodb_database
target_collection_name: mongodb_collection

+export_to_mongodb: 
td>: your_query.sql
result_connection: ${result_connector_name}
result_settings:
database: ${target_database_name}
table: ${target_collection_name}
mode: [append|replace|truncate|update]



  • No labels