This integration allows you to export TD job results into your existing MongoDB instance. For sample workflows on this, please view Treasure Boxes.

Prerequisites

Reference Architecture

A front-end application collects data into Treasure Data via the Treasure Agent. Treasure Data periodically runs jobs on the data, then writes the job results to your MongoDB collections.

Example 1: Ranking: What are the “Top N of X?”

Every social/mobile application calculates the “top N of X” (e.g., top 5 movies watched today). Treasure Data already handles the raw data warehousing; the “write-to-mongodb” feature also enables Treasure Data to find the “top N” data.

Example 2: Dashboard Application

If you are a data scientist, you need to keep track of a range of metrics every hour/day/month and make them accessible via visualizations. Using this “write-to-mongodb” feature, you can streamline the process and focus on your queries and visualizations of the query results.

You can limit access to your MongoDB instance by using a list of static IPs. Contact support@treasuredata.com if you require a static IP.

Define your Query

Navigate to Data Workbench > Queries.
Select New Query.
Run the query to validate the result set.


Specify the Result Export Target

 Select Export Results.

You can select an existing authentication or create a new authentication for the external service to be used for output. Choose one of the following:

Use Existing Integration


Create a New Integration


Specify information for Export to MongoDB.


Field NameDescription
HostThe hostname or IP address of the remote server. (You can add more than one IP address, depending on your MongoDB setup.)
UsernameUsername to connect to the remote database.
PasswordPassword to connect to the remote database.
Use TLS?

Check this box to connect using TLS (SSL).

Bypass certificate validations?Check this box to bypass all certificate validations.
Database nameThe name of the database to which you are transferring data (e.g., your_database_name).
Table NameThe name of the collection to which you are transferring data.
Mode

Append - Add to the existing records in the database. This mode is atomic.

Replace - Replace the existing records with the query results. This mode is atomic.

Truncate - Truncate the existing records. This mode is atomic.

Update - a row is inserted unless it can cause a duplicate value in the columns specified in the “keys” parameter


Create an Activation Using an Integration

Integration Parameters for MongoDB

Field NameDescription
Database nameThe name of the database to which you are transferring data (e.g., your_database_name).
Table NameThe name of the collection to which you are transferring data.
Mode

Append - Add to the existing records in the database. This mode is atomic.

Replace - Replace the existing records with the query results. This mode is atomic.

Truncate - Truncate the existing records. This mode is atomic.

Update - a row is inserted unless it can cause a duplicate value in the columns specified in the “keys” parameter

Keys

Comma-separated list of keys. If the values for all keys match an existing record, that record will be replaced.

Example of a Query

SELECT * FROM my_table

Optionally Schedule the Query Export Jobs

You can use Scheduled Jobs with Result Export to periodically write the output result to a target destination that you specify.

Optionally Configure Export Results in Workflow

Within Treasure Workflow, you can specify the use of this data connector to export data.

Learn more at Using Workflows to Export Data with the TD Toolbelt.

Example Workflow for MongoDB

#Example Worflow
_export:
result_connector_name: mongodb_connector
target_database_name: mongodb_database
target_collection_name: mongodb_collection

+export_to_mongodb: 
td>: your_query.sql
result_connection: ${result_connector_name}
result_settings:
database: ${target_database_name}
table: ${target_collection_name}
mode: [append|replace|truncate|update]