Luigi Workflow Engine with Treasure Data

Luigi is a powerful, easy-to-use open-source framework for building data pipelines with Python. It handles dependency resolution, workflow management, visualization etc.

Luigi helps to build the data pipeline, typically associated with long-running batch processes. You want to chain many tasks together, automate them, and gracefully handle any failures.

Luigi pipelines automatically handle dependency resolution, checkpointing and failure recovery, parallel execution, command line integration, and much more. Pipelines are expressed as simple, easy-to-read Python that can be reused anywhere.

Luigi-TD is a helper library to make it easier to issue Treasure Data queries from Luigi tasks. This will allow you to create complex pipelines on top of TD easily.


Last modified: Feb 10 2015 01:21:12 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.