Legacy Bulk Import Tips And Tricks

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

This article describes some tips and tricks for bulk import.

Solving Error: There Was a Problem Accessing the Remote XML Resource

This error may have occurred before v0.16.7 versions.

When you encounter the following error in td import:jar_update, use one of the following solutions to resolve the error.

Error message:

Error: There was a problem accessing the remote XML resource
'http://central.maven.org/maven2/com/treasuredata/td-import/maven-metadata.xml'
(TreasureData::Command::UpdateError: An error occurred when fetching
from 'http://central.maven.org/maven2/com/treasuredata/td-import/maven-metadata.xml'.)

Solution 1: Update TD Toolbelt Version

The error is solved in v0.16.8 or later versions.

Solution 2: Set Variable

Setting an environment variable can avoid the error.

$ export TD_TOOLBELT_JARUPDATE_ROOT=https://repo1.maven.org

Using a Proxy Server

If you cannot upload your data, verify that your network is using a proxy. You can set the proxy by setting the environment variables:

Increasing Performance through Parallelism

td import:auto supports two options to tune parallelism: --parallel and --prepare-parallel.

See the TD Toolbelt Command Reference for the full syntax reference.

$ td import:auto session name <files...>
--parallel NUM
--prepare-parallel NUM

• Parallel specifies how many threads are to be used for uploading the data. If you observe that the bulk import tool is not saturating your network, you can increase the value of the --parallel option. Default is 2, maximum is 8.
• Prepare parallel specifies the number of threads to be used to compress the data locally. Normally, this number should match the number of CPU cores on your machine. Default is 2, maximum is 96.

Specifying a Time Column for Maximum Query Performance

Don’t specify ‘0’ if you don’t have a time column. Treasure Data partitions the data by time by default. See Data Partitioning. It is recommended to always specify the time column, or specify the current time.

Selecting Enable or Disable Auto Jar_Update

The option to select Enable/Disable auto jar_update can be included in td v0.11.2 and later versions

An environment variable hook: TD_TOOLBELT_JAR_UPDATE.

JAR auto-update is enabled by default or enabled if the variable is 1:

$ td import:prepare
$ TD_TOOLBELT_JAR_UPDATE=1 td import:prepare

JAR auto-update is disabled then variable is set to 0:

$ TD_TOOLBELT_JAR_UPDATE=0 td import:prepare

but this setting does not affect td import:jar_update, which always updates the JAR file.

Confirming Time Zone

The bulk import tool uses a TZ environment variable. If you think your bulk import time zone is wrong, check your TZ environment variable.

Encoding Shift_JIS

When you encode shift_jis you should set the encoding option to '-e Windows-31J'.

Using Time-Format

If you want to assign the data source’s time format to bulk import, you can use --time-format in accordance with the following correspondence table.