...
pytd.query_engine.QueryEngine
Examples
| Code Block | ||
|---|---|---|
| ||
>>> import pytd.pandas_td as td
>>> con = td.connect(apikey=apikey, endpoint="https://api.treasuredata.com")
>>> engine = td.create_engine("presto:sample_datasets") |
...
Subclasses of Configurable must call the __init__() method of Configurable before doing anything else and using super():
| Code Block | ||
|---|---|---|
| ||
class MyConfigurable(Configurable):
def __init__(self, config=None):
super(MyConfigurable, self).__init__(config=config)
# Then any other code you need to finish initialization.
|
...
List databases in the form of pandas.DataFrame.
| Code Block | ||
|---|---|---|
| ||
%td_databases [<database_name_pattern>] |
...
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %td_databases sample
Out[2]:
name count permission created_at updated_at
0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 348124 administrator 2019-01-23 05:48:11+00:00 2019-01-23 05:48:11+00:00
1 yyyyyyyyy 0 administrator 2017-12-14 07:52:34+00:00 2017-12-14 07:52:34+00:00
2 zzzzzzzzzzzzz 0 administrator 2016-05-25 23:12:06+00:00 2016-05-25 23:12:06+00:00
...
In [3]: %td_databases sample
Out[3]:
name count permission created_at updated_at
0 sampledb 2 administrator 2014-04-11 22:29:38+00:00 2014-04-11 22:29:38+00:00
1 sample_xxxxxxxx 2 administrator 2017-06-02 23:37:41+00:00 2017-06-02 23:37:41+00:00
2 sample_datasets 8812278 query_only 2014-10-04 01:13:11+00:00 2018-03-16 04:59:06+00:00
...
|
...
List tables in databases.
| Code Block | ||
|---|---|---|
| ||
%td_tables [<table_identifier_pattern>] |
...
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %td_tables
Out[2]:
db_name name count estimated_storage_size last_log_timestamp created_at
0 xxxxx_demo_aa customer_test 70 1047 2018-02-05 06:20:32+00:00 2018-02-05 06:20:24+00:00
1 xxxxx_demo_aa email_log 0 0 1970-01-01 00:00:00+00:00 2018-02-05 07:19:57+00:00
2 yy_wf topk_similar_items 10598 134208 2018-04-16 09:23:57+00:00 2018-04-16 09:59:48+00:00
...
In [3]: %td_tables sample
Out[3]:
db_name name count estimated_storage_size last_log_timestamp created_at
0 xx_test aaaaaaaa_sample 0 0 1970-01-01 00:00:00+00:00 2015-10-20 17:37:40+00:00
1 sampledb sampletbl 2 843 1970-01-01 00:00:00+00:00 2014-04-11 22:30:08+00:00
2 zzzz_test_db sample_output_tab 4 889 2018-06-06 08:26:20+00:00 2018-06-06 08:27:12+00:00
...
|
...
List job activities in an account.
| Code Block | ||
|---|---|---|
| ||
%td_jobs |
Returns
Return type
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %td_jobs
Out[2]:
status job_id type start_at query
0 error 448650806 hive 2019-04-12 05:33:36+00:00 with null_samples as (\n select\n id,\n ...
1 success 448646994 presto 2019-04-12 05:23:29+00:00 -- read_td_query\n-- set session distributed_j...
2 success 448646986 presto 2019-04-12 05:23:27+00:00 -- read_td_query\n-- set session distributed_j...
...
|
...
This magic pushes all table names in a specified database into the current namespace.
| Code Block | ||
|---|---|---|
| ||
%td_use [<database_name>] |
...
<database_name> (string) – Database name.
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython In [2]: %td_use sample_datasets INFO: import nasdaq INFO: import www_access In [3]: nasdaq # describe table columns in the form of DataFrame Out[3]: <pytd.pandas_td.ipython.MagicTable at 0x117651908> |
...
Get job result.
| Code Block | ||
|---|---|---|
| ||
%td_job [--pivot] [--plot] [--dry-run] [--verbose]
[--connection <connection>] [--dropna] [--out <out>]
[--out-file <out_file>] [--quiet] [--timezone <timezone>]
job_id |
...
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %td_job 451709460 # select * from sample_datasets.nasdaq limit 5
Out[2]:
symbol open volume high low close
time
1992-08-25 16:00:00 ATRO 0.0 3900 0.7076 0.7076 0.7076
1992-08-25 16:00:00 ALOG 0.0 11200 11.0000 10.6250 11.0000
1992-08-25 16:00:00 ATAX 0.0 11400 11.3750 11.0000 11.0000
1992-08-25 16:00:00 ATRI 0.0 5400 14.3405 14.0070 14.2571
1992-08-25 16:00:00 ABMD 0.0 38800 5.7500 5.2500 5.6875 |
...
Run a Hive query.
| Code Block | ||
|---|---|---|
| ||
%%td_hive [<database>] [--pivot] [--plot] [--dry-run] [--verbose]
[--connection <connection>] [--dropna] [--out <out>]
[--out-file <out_file>] [--quiet] [--timezone <timezone>]
<query> |
...
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %%td_hive
...: select hivemall_version()
...:
Out[2]:
_c0
0 0.6.0-SNAPSHOT-201901-r01 |
...
Run a Presto query.
| Code Block | ||
|---|---|---|
| ||
%%td_presto [<database>] [--pivot] [--plot] [--dry-run] [--verbose]
[--connection <connection>] [--dropna] [--out <out>]
[--out-file <out_file>] [--quiet] [--timezone <timezone>]
<query> |
...
pandas.DataFrame
Examples
| Code Block | ||
|---|---|---|
| ||
In [1]: %load_ext pytd.pandas_td.ipython
In [2]: %%td_presto
...: select * from sample_datasets.nasdaq limit 5
...:
Out[2]:
symbol open volume high low close
time
1989-01-26 16:00:00 SMTC 0.0 8000 0.4532 0.4532 0.4532
1989-01-26 16:00:00 SEIC 0.0 163200 0.7077 0.6921 0.7025
1989-01-26 16:00:00 SIGI 0.0 2800 3.9610 3.8750 3.9610
1989-01-26 16:00:00 NAVG 0.0 1800 14.6740 14.1738 14.6740
1989-01-26 16:00:00 MOCO 0.0 71101 3.6722 3.5609 3.5980
|
magics= {'cell': {'td_hive': 'td_hive', 'td_presto': 'td_presto'}, 'line': {'td_job': 'td_job'}}
registered= True
pytd.pandas_td.ipython.load_ipython_extension
pytd.pandas_td.ipython.load_ipython_extension(ipython)[source]