Bulk Import from MySQL

This article explains how to import data from MySQL to Treasure Data.

Table of Contents

Install Bulk Loader

First, please install the toolbelt, which includes bulk loader program, on your computer.

Downloads

After the installation, the td command will be installed on your computer. Open up the terminal and type td to execute the command. Also, please make sure you have java as well. Execute td import:jar_update to download the up-to-date version of our bulk loader:

$ td
usage: td [options] COMMAND [args]
$ java
Usage: java [-options] class [args...]
$ td import:jar_update
Installed td-import.jar 0.x.xx into /path/to/.td/java

Importing data from MySQL

Run the following commands to upload data from MySQL.

$ td db:create my_db
$ td table:create my_db my_tbl
$ td import:auto \
  --auto-create my_db.my_tbl \
  --format mysql \
  --db-url jdbc:mysql://my_sql_host/my_sql_db \
  --db-user my_user \
  --db-password my_pass \
  --time-column date_time \
  my_sql_tbl
Untitled-3
Because `td import:auto` executes MapReduce jobs to check the invalid rows, it'll take at least 1-2 minutes.

In this example, we assumed that:

  • the bulk import will read the input data from the my_sql_tbl on the MySQL database specified by the ‘--db-’ prefixed options
  • the bulk import session is created automatically by our request using the --auto-create option.
    When using the --auto-create option we are required to provide the database and table name in in the format indicated. The session name will be ‘my_db.my_tbl’.
    Alternatively one can create the bulk import session with a name of choice, using the td import:create command (for details see Bulk Import Internals, Import Step-by-Step).
  • the time field is called “date_time” and is provided with the --time-column option

These options can be used to specify the access to the MySQL database:

MySQL specific options:
--db-url URL                     JDBC connection URL
--db-user NAME                   user name for MySQL account
--db-password PASSWORD           password for MySQL account

For further details, check the following pages:


Last modified: Aug 03 2015 00:01:48 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.