Review these Treasure Workflow prerequisites and limitations to understand what you can and cannot accomplish with workflows.

Prerequisites

Before you start creating a Treasure Workflow, you must have a database and table in Treasure Data.

Limitations

Digdag Syntax Limitations

Treasure Workflow currently allows for most of the functionality that Digdag allows, with the following exceptions:

  • sh> for running shell scripts

  • rb> for running ruby scripts

  • embulk> for running arbitrary Embulk jobs (but you can use td_load> for importing bulk data into Treasure Data)

  • download_file: parameter with the td> and other operators for downloading files locally. Instead, use the Treasure Data result export functionality

py:> is the only supported custom script. 

Treasure Workflow Service Limits

TD Workflow is designed to provide a scalable and flexible solution for you to manage your data pipelines in a cloud-hosted environment.

Limits of Use

To ensure optimal and fair processing, the following limits exist:

  • Maximum of 10 tasks running concurrently, per account. All other tasks get queued and are issued on a first-come-first-served basis.

  • 8,000 Saved Workflows. Over 8,000, and not all workflows will load within our Workflow UI. They are still accessible via CLI.

  • The maximum total tasks in an attempt are 1000.

  • Maximum attempts per account are:

    • US - 200
    • Tokyo - 200
    • EU - 200
    • All other regions - 100
  • Period of time until a running task is killed is 1 day.

  • Period of time until a running attempt is killed is 7 days.

  • The maximum size of response of http> is 262144 bytes(256 KB)
  • The maximum size of response of http_call> is 524288 bytes(512 KB)

Advised Limits of Use

For better performance of the TD Console Workflow, try to stay below the following limits:

  • 200 Saved Workflows

  • 400 Tasks in a Workflow

These limits are subject to change if there are changes to the configuration of Treasure Data capacity.

  • No labels