Treasure Workflow comes with Workflow Console which allows you to quickly understand your workflow activities, manage workflow executions and create new workflows.
In this topic, we will first explain how to check your account's workflow. After that, let's define your first workflow.
Use the workflow selector at the top of the Workflows page, to choose the type of workflows you want to view:
User-Defined Workflows — These workflows are created and managed by the users. The workflows perform custom tasks and can be customized for the specific needs and requirements of the user. These workflows provide a high degree of flexibility in the logic and in the definition of processing parameters. An example of a user-defined workflow would be a defined set of activations that need to occur on an activation event.
System Workflows — These workflows are pre-defined and are created by the system or by internal services and applications. Users cannot create or modify these workflows, but they can access them through the user interface. Examples of system workflows are:
- Predictive Scoring
- A/B Test
- Activation
- Master Segment
When you first open the workflow landing page, Treasure Data displays projects in a tree structure with the associated workflow(s). To search for a workflow, you can open each project and view the workflows. Optionally, you can search for a workflow.
In the upper right, select the magnifying icon.
Enter your search.
In the previous example, we searched using the term "one". In the last entry, we see a nested workflow that contains the characters in the workflow name.
Select the sort icon to the right of the Name column.
Select Sort workflow across projects by name.
Select either Sort Ascending or Sort Descending to refresh the page.
Treasure Data displays a flat structure of projects and their associated workflows.
| Status Name | Action | Example Description |
|---|---|---|
| Update Date | Identify the first date that the workflow was run. | This segment was first run on April 8, 2019, at 1:08 pm. |
| Last 5 Runs | Quickly review the success of the last 5 workflow run attempts. | All of the last 5 workflow runs have failed. |
| Last Attempt | Confirm the last attempted workflow run. | The last attempted workflow run was on July 17, 2022, 5:40 pm. |
| Schedule | For repeating workflows, the schedule displays. | Daily at 12:00 am UTC time zone. |
| Next Attempt | The next time Treasure Data will attempt to run the workflow. | Today 5:00 pm. |
Select a workflow row to show its details: run history, project revisions and secrets.
The default workflow detail is Run History. Each workflow run includes the run status, session, last attempt, duration, and revision.
Review the workflow's project revision dates and times and who made the changes. Select a revision link to review the code.
A pencil icon indicates that there is code for the .dig file. A magnifying icon indicates that there isn't any code.
Select the secret link to review secret details. Select the "+" icon to create a new secret. Secrets are secured values like passwords, access tokens and secret URLs.
By clicking some workflow row in Run History tab, you can see its execution details:
This view shows workflow tasks, their duration and logs. You can view task logs by clicking the "WF" button in the "Logs" column.
From the Workflow landing page, you can create a workflow composed from several tasks and schedule that workflow to run regularly. Follow these steps to create, edit, and run a simple workflow.
You must have a database and table in Treasure Data to complete this workflow.
At first, click "New User-Defined Workflow" button:
In the "Create Workflow" pane, enter a workflow name. Project name is defaulted to the workflow name, keep it or select a different one.
You can choose a template by "Workflow Template" select box. Select "Simple Import and Saved Query" for this example. Note that you can scroll this list.
Finally, click "New Workflow" to create the workflow.
After you create a workflow, an editor displays:
This is Workflow Project Editor where you can make changes to the code.
Each code section includes instructions marked with a hashtag (#). These lines are comments, that are ignored by Treasure Workflow.
You cannot start editting in the Workflow Project Editor. Click "Edit Files" button first before changing file contents.
Then, replace td.database value with the name of your database in the _export section (dbname.table by default). This is the database that your query runs in. For example, if your database is your_database, change it like:
_export:
td:
database: your_databaseTo save contents and finish editing, click the "Save & Commit" button. This button saves all files to the Treasure Workflow server and create a new project revision (version).
Treasure Workflow uses a YAML syntax. Pay close attention to the indentation and use of colons.
Treasure Data uses the td_load operator to load data from storage, other databases, or services. In this example, we’ll reference Data Connector’s Unique ID.
To find a unique ID, follow these steps:
- In the TD Console, navigate to Integrations Hub > Sources.
- Select a connector and then select the more menu (…).
- Select Copy Unique ID.
Open your workflow details and click "Launch Project Editor":
At +task1, replace the td_load>: value with the copied Unique ID like this:
+task1:
td_load>: 123456789At +task2, update the saved query name like this:
+task2:
td_run>: input_your_saved_query_name_hereFinally, click "Save & Commit" to save the workflow.
You can run a workflow right away or create a schedule. To run the created workflow now, click the "New Run" button:
It starts new workflow execution.
To monitor workflow execution, open "Run History" tab of the workflow details and click running workflow. It opens execution details view and you can monitor execution status.