Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
linenumberstrue
SELECT id, name, address FROM my_table

...


Excerpt
hiddentrue

Use result_output_redirect

...

By setting result_output_redirect='true'

...

, you can

...

enhance the performance of a Presto job.

Code Block
linenumberstrue
-- set session result_output_redirect='true'
select * FROM my_table

With this query hint, our Presto produces the query results in parallel and can skip the JSON conversion process at the coordinator.

...


Note

In the following situations, the query hint dos not work:

  • When a query includes ORDER BY

  • When a query is CREATE TABLE or INSERT TABLE

How Does it Work?

td-worker processes data in a single thread as follows:

  • downloading query results

  • transforming JSON response from the presto coordinator into msgpack.gz format

  • uploading them to S3

Queries that Produce a Large Volume of Query Results

If the query result has 10GB or more data, td-presto is waiting for query result retrieval by td-worker even though the query result is ready. td-worker is a single thread Ruby process and is not designed to read such large query results; currently, td-worker interleaves various data processing in a single thread, such as downloading query results, transforming JSON responses from the presto coordinator into msgpack.gz format, and uploading them to S3. It cannot fully utilize machine resources, therefore, Treasure Data recommends that you redirect the query result to S3 to make increase the performance.