Visit our new documentation site! This documentation page is no longer updated.

Running Presto & Hive Jobs in Coordination

Table of Contents


In this tutorial you create a workflow that runs presto and hive queries in coordination with one another.


Introductory Tutorial

If you haven’t already, start by going through the TD Workflows introductory tutorial

We will use the workflow project downloaded in the above linked tutorial below.


Review workflow

As a reminder, here is the workflow from the introductory tutorial that we will be modifying.

timezone: UTC

  daily>: 07:00:00

    database: workflow_temp

  td>: queries/daily_open.sql
  create_table: daily_open

  td>: queries/monthly_open.sql
  create_table: monthly_open

Run workflow as is

First, go to your Treasure Data jobs page, so you can follow along as you run the following workflows. You can access the jobs page.

Now, go into the nasdaq_analysis directory, and run the following command.

$ td wf run --all
Why should you include the `--all`? It's because you've probably run this workflow before. As a result, TD Workflows automatically remembers that it has run successfully, and thus shouldn't need to be run again. You can overwrite this condition by including this condition.

You should see the following job runs in your account.

two presto queries

Modify so the depenent task executes a hive job

Having a query run on hive is as simple as adding an extra parameter. Modify your workflow file with the following addition of engine: hive as a parameter to +task2.

  td>: queries/monthly_open.sql
  create_table: monthly_open
  engine: hive

Run the query of the modified workflow

$ td wf run --all

Presto & hive queries in sync

That’s it! It’s now simple to run presto and hive jobs in coordination on Treasure Data using Workflows.


If you have any feedback we welcome hearing your thoughts on our TD Workflows ideas forum.

Also, if you have any ideas or feedback on the tutorial itself, we’d welcome them here!

Last modified: Feb 24 2017 09:41:25 UTC

If this article is incorrect or outdated, or omits critical information, let us know. For all other issues, access our support channels.