Embulk Overview

This article explains how to import data using Treasure Data’s open-source bulk data loader, which is called Embulk.

embulk

Table of Contents

Prerequisites

  • Basic knowledge of Treasure Data.
  • Basic Knowledge of Embulk

What is Embulk?

Embulk is a open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.

How to install Embulk

Linux & Mac & BSD

Embulk is a Java application. Please make sure that Java is installed.

Following 4 commands install embulk to your home directory:

curl --create-dirs -o ~/.embulk/bin/embulk -L "http://dl.embulk.org/embulk-latest.jar"
chmod +x ~/.embulk/bin/embulk
echo 'export PATH="$HOME/.embulk/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Windows

Embulk is a Java application. Please make sure that Java is installed.

You can download embulk.bat using this command on cmd.exe or PowerShell.exe:

PowerShell -Command \
  "& {Invoke-WebRequest http://dl.embulk.org/embulk-latest.jar -OutFile embulk.bat}"

How to install Treasure Data plugin

You can use plugins to load data from/to various systems and file formats. Here is the list of publicly released plugins: list of plugins by category.

The following command installs embulk-output-td plugin, which imports records to Treasure Data.

embulk gem install embulk-output-td

Use Cases

Tips

How to use a proxy server

If you cannot upload, please first check if your network is using a proxy. You can set the proxy by command line option:

Linux:
  embulk -J-Dhttp.proxyHost=xxxx -J-Dhttp.proxyPort=xxxx -J-Dhttp.proxyUser=xxxx -J-Dhttp.proxyPassword=xxxx run config.yml
Windows:
  embulk.bat "-J-Dhttps.proxyHost=xxxx" "-J-Dhttps.proxyPort=xxxx" "-J-Dhttp.proxyUser=xxxx" "-J-Dhttp.proxyPassword=xxxx" run config.yml
Or,
  "java"  -Dhttps.proxyHost="host" -Dhttps.proxyPort="port" -jar embulk.bat run config.yml

More details

For more info about embulk, please see the following:


Last modified: Jun 28 2016 00:50:14 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.