This article explains how to import data from MySQL database to Treasure Data using embulk-input-mysql input plugin.
Basic knowledge of Treasure Data.
Basic knowledge of Embulk.
Embulk is a Java application. Make sure that Java is installed.
Follow the instructions in Installing Bulk Data Import.
Embulk and embulk-output-td plugin installed on your machine.
To install embulk-input-mysql plugin, run the following command:
embulk gem install embulk-input-mysqlUsing your favorite text editor, create Embulk config file (for eg:seed.yml) defining input(MySQL) and output(TD) parameters. Example:
in:
type: mysql
host: localhost
port: 3306
user: username
password: password
database: mysql_db
table: my_table
select: 'col1, col2, datecolumn'
where: col4 != 'a'
out:
type: td
apikey: xxxxxxxxxxxx
endpoint: api.treasuredata.com
database: dbname
table: tblname
time_column: datecolumn
mode: replace
default_timestamp_format: '%d/%m/%Y'For further details about additional parameters and other examples are available, refer Embulk Input MySQL.
Embulk guess option uses seed.yml to read the target file and automatically guesses the column types/settings and creates a new file load.yml with this information.
embulk guess seed.yml -o load.ymlYou can preview the data using embulk preview load.yml command. If any of the column types or data seems incorrect you may edit load.yml file directly and preview again to verify. If guess option doesn’t yield satisfactory results, you may change parameters in load.yml according to your requirement.
Issue the import job by running the following command:
embulk run load.ymlIt may take few mins to hours for the job to complete, depending on the size of the data.