Review these Treasure Workflow prerequisites and limitations to understand what you can and cannot accomplish with workflows.
Before you start creating a Treasure Workflow, you must have a database and table in Treasure Data.
Digdag Syntax Limitations
Treasure Workflow currently allows for most of the functionality that Digdag allows, with the following exceptions:
sh>for running shell scripts
rb>for running ruby scripts
embulk>for running arbitrary Embulk jobs (but you can use
td_load>for importing bulk data into Treasure Data)
download_file:parameter with the
td>and other operators for downloading files locally. Instead, use the Treasure Data result export functionality
py:> is the only supported custom script.
Treasure Workflow Service Limits
TD Workflow is designed to provide a scalable and flexible solution for you to manage your data pipelines in a cloud-hosted environment.
Limits of Use
To ensure optimal and fair processing, the following limits exist:
Maximum of 10 tasks running concurrently, per account. All other tasks get queued and are issued on a first-come-first-served basis.
8,000 Saved Workflows. Over 8,000, and not all workflows will load within our Workflow UI. They are still accessible via CLI.
The maximum total tasks in an attempt are 1000.
Maximum attempts per account are:
- US - 200
- Tokyo - 200
- EU - 200
- All other regions - 100
Period of time until a running task is killed is 1 day.
Period of time until a running attempt is killed is 7 days.
- The maximum size of response of http> is 262144 bytes(256 KB)
- The maximum size of response of http_call> is 524288 bytes(512 KB)
Advised Limits of Use
For better performance of the TD Console Workflow, try to stay below the following limits:
200 Saved Workflows
400 Tasks in a Workflow
These limits are subject to change if there are changes to the configuration of Treasure Data capacity.