# Workflow Commands

Manage Treasure Data workflows powered by Digdag.

Alias
`tdx wf` is an alias for `tdx workflow`

## Commands

### Navigation & Discovery

| Command | Description |
|  --- | --- |
| `use <project>` | Set default workflow project |
| `projects [pattern]` | List workflow projects |
| `workflows [project]` | List workflows in a project |
| `run [project.workflow]` | Run a workflow (interactive selector if omitted) |
| `sessions [project]` | List workflow sessions |
| `attempts [project]` | List workflow attempts |


### Project Sync

| Command | Description |
|  --- | --- |
| `pull <project>` | Pull project to local folder |
| `push [source-dir]` | Push local changes to TD |
| `clone [source-dir] --name <name>` | Clone project to a new name |
| `delete <project>` | Delete a project |


### Secrets

| Command | Description |
|  --- | --- |
| `secrets list [project]` | List secret keys |
| `secrets set [project] KEY=VALUE` | Set a secret |
| `secrets delete [project] <key>` | Delete a secret |


## Typical Usage


```bash
# 0. List available workflow projects
tdx wf projects

# 1. Pull a workflow project to local folder
#    (this also sets project context)
tdx wf pull myproject
# Creates: workflows/myproject/

# 2. Edit workflow files locally
# Edit workflows/myproject/main.dig with your text editor

# 3. Push changes back to TD
tdx wf push

# 4. Run a workflow (interactive selector shows available workflows)
tdx wf run

# 5. Monitor sessions and attempts (context already set by pull)
tdx wf sessions
tdx wf attempts
```

The pull/push workflow enables version control and code review for workflow definitions:


```bash
# Work from any subdirectory - push finds tdx.json automatically
cd workflows/myproject
git add . && git commit -m "Update ETL schedule"
tdx wf push
```

## How It Works


```mermaid
flowchart LR
    Local["workflows/myproject/*.dig"]
    TD[Treasure Data]

    Local -->|"tdx wf push"| TD
    TD -->|"tdx wf pull"| Local
```

The `tdx wf` commands help you manage workflow projects as local files. You can `pull` existing projects from Treasure Data, edit them locally, and `push` changes back.

Session Context
The default project is stored in your session context (same as `tdx use`). Running `tdx wf pull` also sets the default project automatically. Use `tdx use` to view all current session settings.

## List Projects and Workflows

The list commands display results in a compact list format by default, with status icons and relevant metadata.


```bash
# List all workflow projects (default: 100)
tdx wf projects

# Filter projects by pattern
tdx wf projects "cdp_segment_*"

# Fetch more projects with --limit
tdx wf projects --limit 200

# Output as table format
tdx wf projects --format table

# List all workflows
tdx wf workflows

# List workflows for specific project
tdx wf workflows myproject

# Filter workflows by project pattern
tdx wf workflows "lda*"
```

**Output format:**

- Projects: `📁 project_name` (with `(system)` label for system workflows)
- Workflows: `📋 workflow_name (project_name)`


When results may be truncated, a hint is shown: `(use --limit to show more)`

## Sessions and Attempts

Sessions and attempts are displayed with status icons indicating their state.


```bash
# List all workflow sessions
tdx wf sessions

# List sessions for specific project
tdx wf sessions myproject

# List sessions for specific workflow
tdx wf sessions myproject.daily_etl

# Filter sessions by status
tdx wf sessions --status error
tdx wf sessions --status running

# Filter sessions by time range
tdx wf sessions --from "2025-01-01" --to "2025-01-31"

# Fetch more sessions
tdx wf sessions --limit 200

# List all attempts
tdx wf attempts

# List attempts including retried
tdx wf attempts --include-retried
```

### Interactive Session Navigation

In interactive mode (terminal), `tdx wf sessions` displays a cursor-based selector allowing you to navigate to a session's timeline view:


```bash
# View sessions interactively
tdx wf sessions myproject

# Use arrow keys to navigate, Enter to view timeline, Esc to exit
```

**How it works:**

1. Sessions are displayed in timeline format
2. An interactive selector appears with sessions listed
3. Use ↑↓ arrow keys to navigate
4. Press Enter to view the selected session's timeline
5. Press Esc to exit and view the static session list


**Non-interactive mode:**

- When piping output: `tdx wf sessions | grep "success"`
- When using `--format`: `tdx wf sessions --format json`
- When writing to file: `tdx wf sessions --output sessions.txt`


**Status icons:**

- ✅ success
- ❌ error
- 🔄 running
- 🛑 canceling
- ⏳ pending


**Output format:**

- Sessions: `status_icon project.workflow session_time status`
- Attempts: `status_icon project.workflow #attempt_id session_time status`


## Attempt Management

The `attempt` command provides subcommands for managing workflow attempts:


```bash
tdx wf attempt <attempt-id> [action] [args...]
```

### Show Attempt Details


```bash
# Show specific attempt details
tdx wf attempt 67890
```

### View Tasks


```bash
# Show tasks for an attempt
tdx wf attempt 67890 tasks

# Show tasks including subtasks
tdx wf attempt 67890 tasks --include-subtasks
```

### View Logs


```bash
# View logs with interactive task selector
tdx wf attempt 67890 logs

# View logs for a specific task
tdx wf attempt 67890 logs +step1

# List available log files as JSON
tdx wf attempt 67890 logs --json
```

When no task name is provided, an interactive selector will display available tasks with their log file sizes.

### Kill Attempt


```bash
# Kill a running attempt (with confirmation)
tdx wf attempt 67890 kill

# Kill with reason and skip confirmation
tdx wf attempt 67890 kill --reason "manual stop" -y
```

### Timeline Visualization

View task execution timeline as an ASCII Gantt chart, showing when each task started, its duration, and current status with colored bars.


```bash
# Interactive session selector
tdx wf timeline

# Show latest session for specific workflow (uses session context for project)
tdx wf use myproject
tdx wf timeline daily_etl

# Filter by project and workflow
tdx wf timeline myproject.daily_etl

# Direct navigation using session or attempt ID
tdx wf timeline --session-id 12345
tdx wf timeline --attempt-id 67890

# Follow running attempt with live updates
tdx wf timeline daily_etl --follow

# JSON output for scripting
tdx wf timeline myproject.daily_etl --format json
```

### Direct Timeline Navigation

You can jump directly to a specific session or attempt using ID options:


```bash
# Show timeline for a specific session (uses last attempt)
tdx wf timeline --session-id 12345

# Show timeline for a specific attempt
tdx wf timeline --attempt-id 67890

# Combine with --follow for live updates
tdx wf timeline --attempt-id 67890 --follow
```

**Option priority:**

1. `--attempt-id` - Uses the attempt ID directly (highest priority)
2. `--session-id` - Fetches the session and uses its last attempt
3. `[project.workflow]` - Interactive selector or auto-select latest


If both `--attempt-id` and `--session-id` are provided, `--attempt-id` takes precedence.

**Example Output:**


```
Attempt #164700627 - success - 5m 23s
                                 2026-01-03   14:15:00    14:16:20    14:17:40    14:19:00    14:20:20
✅ +extract_users                  (1m 20s)│───────────────────────                                          │
✅ +transform_data                 (2m 45s)│            ─────────────────────────────────                    │
✅ +load_to_warehouse              (3m 10s)│                                    ───────────────────────────  │
❌ +send_notification                 (5s)│                                                            ×××   │
✅ +cleanup                          (45s)│                                                               ───│
Total: 5  ✅ 4  ❌ 1  🔄 0  ⏳ 0
```

**Timeline Features:**

- **Colored lines** show task execution periods:
  - ✅ Green lines (─) for successful tasks
  - ❌ Red X marks (×) for failed tasks
  - 🔄 Blue lines (─) for running tasks
  - 🛑 Yellow lines (─) for blocked tasks
  - ⏳ Dim dashes (─) for planned tasks
- **Time axis** with auto-scaled labels (seconds to hours)
- **Task statistics** showing success/error/running counts
- **Live updates** with `--follow` flag (refreshes every 2 seconds)
- **Console URL** link for detailed view in web UI


### Timeline Options

| Option | Description |
|  --- | --- |
| `--session-id <id>` | Show timeline for specific session (uses last attempt) |
| `--attempt-id <id>` | Show timeline for specific attempt |
| `--follow` | Follow/watch timeline for running attempts (auto-refresh every 2s) |
| `--limit <n>` | Limit sessions shown in selector (default: 50) |


### Use Cases

**Monitor running workflows:**


```bash
tdx wf timeline myproject --follow
```

**Analyze completed workflow:**


```bash
tdx wf timeline myproject.nightly_batch
```

**Export timeline data:**


```bash
tdx wf timeline myproject.daily_etl --format json > timeline.json
```

### Retry Attempt


```bash
# Retry attempt
tdx wf attempt 67890 retry

# Retry attempt from specific task
tdx wf attempt 67890 retry --resume-from +step2

# Retry with parameter override
tdx wf attempt 67890 retry --params '{"key":"value"}'

# Force retry without confirmation
tdx wf attempt 67890 retry --force -y
```

## Run Workflow

Start a new workflow run (attempt) directly from the command line.


```bash
# Start a workflow run
tdx wf run myproject.myworkflow

# Start with parameters
tdx wf run myproject.myworkflow --param key1=value1 --param key2=123

# Start with multiple parameters
tdx wf run myproject.myworkflow --param env=production --param date=2025-01-15 --param count=100

# Start with custom session time
tdx wf run myproject.myworkflow --session-time "2025-01-15T10:00:00Z"

# Combine parameters and session time
tdx wf run myproject.myworkflow --param env=staging --session-time "2025-01-15T00:00:00+09:00"
```

### Run Options

| Option | Description |
|  --- | --- |
| `--param <key=value>` | Workflow parameter (repeatable) |
| `--session-time <time>` | Session time in ISO 8601 format (default: now) |


### Parameter Type Detection

Parameter values are automatically parsed:

- Numbers: `--param count=100` → `{"count": 100}`
- Booleans: `--param enabled=true` → `{"enabled": true}`
- Strings: `--param name=hello` → `{"name": "hello"}`
- JSON: `--param config={"a":1}` → `{"config": {"a": 1}}`


## Session Retry

To retry a session (as opposed to a specific attempt), use the `retry` command with the `session:` prefix:


```bash
# Retry entire session
tdx wf retry session:12345

# Retry session from specific task
tdx wf retry session:12345 --from-task +step2

# Retry with parameter override
tdx wf retry session:12345 --params '{"key":"value"}'

# Retry with params from file
tdx wf retry session:12345 --params @params.json
```

## Project Management

### Pull/Push Workflow

The `pull` and `push` commands provide a sync-style workflow for managing workflow projects locally, similar to how `tdx sg` and `tdx journey` work for segments.


```bash
# Pull project to local folder
tdx wf pull myproject                    # Creates workflows/myproject/

# Pull to custom directory
tdx wf pull myproject ./my-workflows

# Pull specific revision
tdx wf pull myproject --revision v1.0.0

# Preview changes without writing
tdx wf pull myproject --dry-run
```

After pulling, your local folder will contain:

- `tdx.json` - Project metadata for sync tracking
- `*.dig` - Workflow definition files
- `*.sql`, `*.py`, etc. - Supporting files



```bash
# Push local changes back to TD
cd workflows/myproject
tdx wf push

# Push from specific directory
tdx wf push ./workflows/myproject

# Push with custom revision
tdx wf push --revision v2.0.0

# Preview changes without pushing
tdx wf push --dry-run
```

The push command will:

1. Compare local files with the remote project
2. Show a summary of changes (new, modified, deleted files)
3. Display diffs for modified files
4. Ask for confirmation before pushing


### Clone Project

The `clone` command creates a copy of a workflow project with a different name. This is useful for:

- Deploying workflows to different environments (dev → staging → prod)
- Creating variations of existing workflows
- Pushing to different TD profiles (cross-profile deployment)



```bash
# Clone from a local directory to a new project name
tdx wf clone workflows/myproject --name myproject-staging

# Clone using session context
tdx wf use myproject
tdx wf clone --name myproject-prod

# Clone with custom revision
tdx wf clone workflows/myproject --name myproject-v2 --revision v2.0.0

# Preview before creating
tdx wf clone workflows/myproject --name myproject-test --dry-run

# Cross-profile deployment
tdx wf clone workflows/myproject --name myproject-prod --profile production
```

The clone command will:

1. Check if the source directory contains valid workflow files
2. Check if a project with the target name already exists
3. Show a warning if the target project exists (will create new revision)
4. Ask for confirmation before creating/updating the project


Difference from Push
- **push** updates the *same* project (based on tdx.json)
- **clone** creates/updates a *different* project with a new name


### Download Project (Legacy)


```bash
# Download workflow project (without tdx.json tracking)
tdx wf download myproject

# Download to specific directory
tdx wf download myproject ./workflows

# Download specific revision
tdx wf download myproject --revision v1.0.0
```

### Upload Project (Legacy)

The `upload` command provides digdag-style push for users who prefer the traditional workflow:


```bash
# Upload workflow project (digdag-style)
cd myproject
tdx wf upload myproject

# Upload with custom revision
tdx wf upload myproject --revision v1.0.0

# Upload from parent directory with path
tdx wf upload ./myproject

# Upload with --name override
tdx wf upload myproject --name production_workflow

# Upload without validation
tdx wf upload myproject --skip-validation
```

### Delete Project


```bash
# Delete workflow project (with confirmation)
tdx wf delete myproject

# Delete by project ID
tdx wf delete 12345

# Delete without confirmation
tdx wf delete myproject -y
```

## Command Options

### List Commands Options

These options apply to `projects`, `workflows`, `sessions`, and `attempts` commands.

| Option | Description |
|  --- | --- |
| `--limit <n>` | Maximum results to return (default: 100, fetches multiple pages if needed) |
| `--format <format>` | Output format: `table`, `json`, `jsonl` (default: list style) |


### Sessions Options

| Option | Description |
|  --- | --- |
| `--status <status>` | Filter by status (running, success, error, blocked, all) |
| `--from <timestamp>` | Start time filter (ISO 8601) |
| `--to <timestamp>` | End time filter (ISO 8601) |


### Attempts Options

| Option | Description |
|  --- | --- |
| `--include-retried` | Include retried attempts |


### Attempt Options

| Option | Description |
|  --- | --- |
| `--include-subtasks` | Include subtasks in tasks output |
| `--reason <text>` | Reason for killing the attempt |
| `--resume-from <task>` | Resume from specific task (retry) |
| `--params <json>` | Override parameters for retry (JSON or @file.json) |
| `--force` | Force retry even if not failed |
| `-y, --yes` | Skip confirmation prompt |


### Session Retry Options

| Option | Description |
|  --- | --- |
| `--from-task <task>` | Resume from specific task |
| `--params <json>` | Override parameters (JSON or @file.json) |


### Pull Options

| Option | Description |
|  --- | --- |
| `--revision <revision>` | Pull specific revision (default: latest) |
| `--dry-run` | Preview changes without writing files |


### Push Options

| Option | Description |
|  --- | --- |
| `--revision <revision>` | Custom revision name (default: timestamp) |
| `--dry-run` | Preview changes without pushing |
| `--skip-validation` | Skip validation of .dig files |


### Clone Options

| Option | Description |
|  --- | --- |
| `--name <name>` | Name for the new project (required) |
| `--revision <revision>` | Custom revision name (default: timestamp) |
| `--dry-run` | Preview without creating the project |
| `--skip-validation` | Skip validation of .dig files |


### Upload Options (Legacy)

| Option | Description |
|  --- | --- |
| `--name <name>` | Project name (default: directory name) |
| `--revision <revision>` | Revision name (default: timestamp) |
| `--skip-validation` | Skip validation of .dig files |


## Secrets Management

Manage secrets for workflow projects. Secrets can be used in workflow configurations (e.g., `${secret:td.apikey}`).

Default Project
Set a default project with `tdx wf use <project>` to omit the project argument from secrets commands.

### List Secrets


```bash
# List all secret keys for a project
tdx wf secrets list myproject

# With default project set (tdx wf use myproject)
tdx wf secrets list

# List secrets by project ID
tdx wf secrets list 12345

# Output as JSON
tdx wf secrets list myproject --json
```

### Set Secret


```bash
# Set a secret for a project
tdx wf secrets set myproject td.apikey=YOUR_API_KEY

# With default project set
tdx wf secrets set td.apikey=YOUR_API_KEY

# Set secret with value containing special characters
tdx wf secrets set myproject DB_URL='postgres://user:pass@host:5432/db'

# Set secret with spaces in value
tdx wf secrets set myproject MESSAGE='Hello World'
```

Shell Quoting
The shell handles quote removal, so use quotes to protect values with special characters or spaces:

- `KEY='my value'` → value is `my value`
- `KEY="value"` → value is `value`
- `"KEY='value'"` → value is `'value'` (quotes preserved)


### Delete Secret


```bash
# Delete a secret from a project (with confirmation)
tdx wf secrets delete myproject td.apikey

# With default project set
tdx wf secrets delete td.apikey

# Delete by project ID
tdx wf secrets delete 12345 db.password

# Delete without confirmation
tdx wf secrets delete myproject td.apikey -y
```

### Secrets Command Reference

| Command | Description |
|  --- | --- |
| `tdx wf secrets list [project]` | List secret keys for a project |
| `tdx wf secrets set [project] KEY=VALUE` | Set a secret for a project |
| `tdx wf secrets delete [project] <key>` | Delete a secret from a project |


Project Resolution
When project is omitted, it uses the default project set via `tdx wf use`.