Compatible Functions

pytd.pandas_td.connect

pytd.pandas_td.connect(apikey=Noneendpoint=None**kwargs)[source]

Create a connection to Treasure Data

Parameters

Returns

Return type

pytd.Client

pytd.pandas_td.create_engine

pytd.pandas_td.create_engine(urlcon=Noneheader=Trueshow_progress=5.0clear_progress=True)[source]

Create a handler for query engine based on a URL.

The following environment variables are used for default connection:

TD_API_KEY API key TD_API_SERVER API server (default: https://api.treasuredata.com)

Parameters

Returns

Return type

pytd.query_engine.QueryEngine

Examples

>>> import pytd.pandas_td as td
>>> con = td.connect(apikey=apikey, endpoint="https://api.treasuredata.com")
>>> engine = td.create_engine("presto:sample_datasets")

pytd.pandas_td.read_td_query

pytd.pandas_td.read_td_query(queryengineindex_col=Noneparse_dates=Nonedistributed_join=Falseparams=None)[source]

Read Treasure Data query into a DataFrame.

Returns a DataFrame corresponding to the result set of the query string. Optionally provide an index_col parameter to use one of the columns as the index, otherwise default integer index will be used.

While Presto in pytd has two options to issue a query, by either tdclient or prestodb, pytd.pandas_td#read_td_query always uses the former to be compatible with the original pandas-td. Use pytd.Client to take advantage of the latter option.

Parameters

Available parameters:

Returns

Query result in a DataFrame

Return type

pandas.DataFrame

pytd.pandas_td.read_td_job

pytd.pandas_td.read_td_job(job_idengineindex_col=Noneparse_dates=None)[source]

Read Treasure Data job result into a DataFrame.

Returns a DataFrame corresponding to the result set of the job. This method waits for job completion if the specified job is still running. Optionally provide an index_col parameter to use one of the columns as the index, otherwise default integer index will be used.

Parameters

Returns

Job result in a dataframe

Return type

pandas.DataFrame

pytd.pandas_td.read_td_table

pytd.pandas_td.read_td_table(table_nameengineindex_col=Noneparse_dates=Nonecolumns=Nonetime_range=Nonelimit=10000)[source]

Read Treasure Data table into a DataFrame.

The number of returned rows is limited by “limit” (default 10,000). Setting limit=None means all rows. Be careful when you set limit=None because your table might be very large and the result does not fit into memory.

Parameters

Returns

Return type

pandas.DataFrame

pytd.pandas_td.read_td

pytd.pandas_td.read_td(queryengineindex_col=Noneparse_dates=Nonedistributed_join=Falseparams=None)[source]

Read Treasure Data query into a DataFrame.

Returns a DataFrame corresponding to the result set of the query string. Optionally provide an index_col parameter to use one of the columns as the index, otherwise default integer index will be used.

While Presto in pytd has two options to issue a query, by either tdclient or prestodb, pytd.pandas_td#read_td_query always uses the former to be compatible with the original pandas-td. Use pytd.Client to take advantage of the latter option.

Parameters

Returns

Query result in a DataFrame

Return type

pandas.DataFrame

pytd.pandas_td.to_td

pytd.pandas_td.to_td(framenameconif_exists='fail'time_col=Nonetime_index=Noneindex=Trueindex_label=Nonechunksize=10000date_format=Nonewriter='bulk_import'**kwargs)[source]

Write a DataFrame to a Treasure Data table.

This method converts the dataframe into a series of key-value pairs and sends them using the Treasure Data streaming API. The data is divided into chunks of rows (default 10,000) and uploaded separately. If upload failed, the client retries the process for a certain amount of time (max_cumul_retry_delay; default 600 secs). This method may fail and raise an exception when retries did not success, in which case the data may be partially inserted. Use the bulk import utility if you cannot accept partial inserts.

Parameters

IPython Magics

Use IPython magics to access to Treasure Data. Start by loading the magics.

In [1]: %load_ext pytd.pandas_td.ipython

pytd.pandas_td.ipyhton.MagicContext