Review these Treasure Workflow prerequisites and limitations to understand what you can and cannot accomplish with workflows.

Prerequisites

Before you start creating a Treasure Workflow, you must have a database and table in Treasure Data.

Limitations

Treasure Workflow currently allows for most of the functionality that Digdag allows, with the following exceptions:

sh> for running shell scripts
rb> for running ruby scripts
embulk> for running arbitrary Embulk jobs (but you can use td_load> for importing bulk data into Treasure Data)
download_file: parameter with the td> and other operators for downloading files locally. Instead, use the Treasure Data result export functionality

py:> is the only supported custom script.

TD Workflow is designed to provide a scalable and flexible solution for you to manage your data pipelines in a cloud-hosted environment.

To ensure optimal and fair processing, the following limits exist:

Maximum of 10 tasks running concurrently, per account. All other tasks get queued and are issued on a first-come-first-served basis.
8,000 Saved Workflows. Over 8,000, and not all workflows will load within our Workflow UI. They are still accessible via CLI.
The maximum total tasks in an attempt are 1000.
Maximum attempts per account are:
- US - 200
- Tokyo - 200
- EU - 200
- All other regions - 100
Period of time until a running task is killed is 1 day.
Period of time until a running attempt is killed is 7 days.
The maximum size of response of http> is 262144 bytes(256 KB)
The maximum size of response of http_call> is 524288 bytes(512 KB)

For better performance of the TD Console Workflow, try to stay below the following limits:

These limits are subject to change if there are changes to the configuration of Treasure Data capacity.