Troubleshooting Incremental Activation

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

This guide helps you diagnose and resolve common issues when using Incremental Activation workflows.

Common Issues

Workflow Fails with "Duplicate ID Key" Error

Symptom: The workflow fails during execution with an error message indicating duplicate values in the id_key column.

Cause: The column specified as id_key contains non-unique values, which prevents accurate delta calculation.

Solution:

1. Verify that your id_key column contains only unique values by running this query:

   SELECT id_key_column, COUNT(*) as count
   FROM your_database.your_table
   GROUP BY id_key_column
   HAVING COUNT(*) > 1

2. If duplicates exist, either:

   • Choose a different column that is guaranteed to be unique (e.g., td_customer_id)
   • Clean your source data to ensure uniqueness
   • Use a composite key approach (contact Support for guidance)

3. Update your workflow's id_key parameter with the unique column name.

Full Export Runs Instead of Delta

Symptom: Every activation exports the entire segment instead of only delta changes.

Possible Causes:

1. First Run: The initial workflow run always performs a full export to establish the baseline.
2. Delta Key Changed: If you modify the id_key or delta_keys parameters, the workflow triggers a full export.
3. History Table Deleted: The workflow's history table was manually deleted or cleared.
4. Workflow Name Changed: Renaming the workflow creates a new workflow instance without history.

Solution:

1. For first run: This is expected behavior. Subsequent runs will export only deltas.
2. For parameter changes: After modifying id_key or delta_keys, one full export is required. Mark this run as expected and monitor subsequent runs.
3. For deleted history: If the history table (in cdp_audience_{id} database) was deleted, it will be recreated on the next run. One full export is required.
4. For workflow rename: Avoid renaming workflows. If necessary, expect one full export after the change.

No Delta Records Sent Despite Segment Changes

Symptom: The segment has new or changed profiles, but the workflow reports 0 records sent to the destination.

Possible Causes:

1. Activation Mappings Not Configured: Your activation_mappings don't include the delta status you expect.
2. Delta Keys Not Capturing Changes: The columns in delta_keys don't include the attributes that changed.
3. ID Key Mismatch: The id_key doesn't match between runs (e.g., using email when some emails changed).

Solution:

1. Review activation mappings: Ensure you have mappings for all expected delta statuses:

   activation_mappings: [
     {"delta_status": "new", "connector_field": "mode", "connector_field_value": "append"},
     {"delta_status": "updated", "connector_field": "mode", "connector_field_value": "append"}
   ]

2. Check delta keys: Add the changed columns to delta_keys:

   delta_keys: ["status", "tier", "email", "phone"]

3. Verify ID key stability: Ensure your id_key column values don't change between runs.

Connector Fails with "Unsupported Mode" Error

Symptom: The workflow completes delta calculation but fails during result export with an error about unsupported mode or operation.

Cause: Your connector doesn't support the connector_field_value specified in your activation mappings.

Solution:

1. Check your connector's documentation for supported operations:

   • Some connectors only support append
   • Some don't support delete or remove
   • Field names vary by connector (mode vs operation vs action)

2. Update your activation_mappings to use supported values:

   Example - Connector without delete support:

   # Remove the delete mapping if not supported
   activation_mappings: [
     {"delta_status": "new", "connector_field": "mode", "connector_field_value": "append"},
     {"delta_status": "updated", "connector_field": "mode", "connector_field_value": "append"}
     # "deleted" mapping removed
   ]

3. For connectors with limited capabilities, consider using a different activation mode or connector.

API Endpoint Error / 404 Not Found

Symptom: Workflow fails immediately with a 404 error or "endpoint not found."

Cause: The api_endpoint parameter doesn't match your Treasure Data site location.

Solution:

Update the api_endpoint to match your TD site:

Example fix:

_export:
  params:
    api_endpoint: "https://integrations-gateway.eu01.treasuredata.com"  # Changed from us01 to eu01

Workflow Fails with "Column Not Found" Error

Symptom: The workflow fails with an error indicating a column specified in id_key or delta_keys doesn't exist.

Cause: The column name is misspelled, or the column doesn't exist in the source table.

Solution:

1. Verify column names in your source table:

   DESCRIBE your_database.your_table

2. Update your workflow parameters with exact column names (case-sensitive):

   id_key: "td_customer_id"  # Exact name from table schema
   delta_keys: ["subscription_status", "loyalty_tier"]  # Exact names

3. If running as an Activation Action, verify the segment output includes the required columns in the Output Mapping.

Incorrect Number of Profiles Activated

Symptom: The number of profiles sent doesn't match expectations (too many or too few).

Possible Causes:

1. Delta Keys Too Sensitive: Too many columns in delta_keys causing most profiles to be marked as "updated."
2. Delta Keys Not Sensitive Enough: Missing important columns in delta_keys causing actual changes to be missed.
3. Mapping Logic Error: Activation mappings don't match intended behavior.

Solution:

1. Test delta sensitivity: Run a test query to see how many profiles would be marked as changed:

   SELECT
     COUNT(*) as total_profiles,
     SUM(CASE WHEN status_changed THEN 1 ELSE 0 END) as changed_profiles
   FROM your_table

2. Adjust delta keys: Add or remove columns based on what changes should trigger re-activation:

   # Too sensitive - every profile change triggers update
   delta_keys: ["last_login", "last_page_view", "session_count"]
   
   # Better - only meaningful changes trigger update
   delta_keys: ["subscription_tier", "email_opt_in"]

3. Review activation logs: Check the workflow execution logs to see the breakdown of delta statuses.

Permission Denied Errors

Symptom: Workflow fails with "permission denied" when accessing databases, tables, or connectors.

Cause: The user running the workflow doesn't have required permissions.

Solution:

1. Workflow permissions: Ensure the workflow has View and Run permissions for activation creators. See Configuring Workflow Permissions.

2. Database permissions: Verify the user has:

   • Read access to the source database and table
   • Write access to the cdp_audience_{id} database (for history tables)

3. Connector permissions: Verify the user can access the result connector authentication.

4. Contact your TD Administrator to grant necessary permissions.

Standalone Workflow Fails - Missing Parameters

Symptom: Standalone workflow fails with errors about missing activation_actions_db, activation_actions_table, or connection parameters.

Cause: When running as a standalone workflow (not as an Activation Action), these parameters must be manually configured.

Solution:

Add all required parameters for standalone mode:

_export:
  params:
    api_endpoint: "https://integrations-gateway.us01.treasuredata.com"
    id_key: "email"
    delta_keys: ["status", "tier"]
    activation_mappings: [
      {"delta_status": "new", "connector_field": "mode", "connector_field_value": "append"}
    ]
    # Required for standalone mode:
    activation_actions_db: "my_database"
    activation_actions_table: "my_table"
    result_connection_name: "my_connector_auth"
    result_connection_settings:
      type: "snowflake"
      database: "TARGET_DB"
      schema: "PUBLIC"
      table: "TARGET_TABLE"
      mode: "append"

Inconsistent Delta Results or State Conflicts

Symptom: Delta calculations produce unexpected results, with incorrect counts of new, updated, or deleted profiles. Multiple workflow runs show inconsistent behavior or conflicting state.

Cause: Multiple workflows are running concurrently on the same source table. Since the table is not locked during operation, concurrent workflows can interfere with each other's delta calculations, leading to state conflicts.

Solution:

1. Review workflow schedules: Check if multiple workflows or activations are scheduled to run at the same time on the same source table.

2. Adjust scheduling: Ensure only one workflow processes a given table at any time:

   • Stagger workflow schedules to run sequentially, not concurrently
   • Add sufficient buffer time between runs to ensure completion
   • Use workflow dependencies if running multiple related workflows

3. Example - Staggered schedule:

   Workflow A on table_1: Runs at 00:00 (takes ~15 minutes)
   Workflow B on table_1: Runs at 00:30 (safe, no overlap)
   Workflow C on table_2: Runs at 00:00 (safe, different table)

4. Monitor execution: Use workflow execution logs to verify workflows are not overlapping.

5. Best practice: If you need to activate the same segment to multiple destinations, create a single Incremental Activation workflow with multiple activation mappings or use sequential activation actions instead of parallel workflows.

Debugging Tips

Enable Verbose Logging

Add logging to your workflow to track delta calculation:

+fetch_incremental_activation_wf:
  http_call>: "${params.api_endpoint}/integration_workflow/workflows/incremental_activation/fetch"
  # ... rest of configuration ...

+log_completion:
  echo>: "Incremental activation completed successfully"

Check History Tables

The workflow creates history tables in the cdp_audience_{id} database. You can query these tables to understand what data was used for delta calculation:

-- View the last activation snapshot
SELECT * FROM cdp_audience_123.incremental_activation_history
LIMIT 100

Test with Small Segments

Before deploying to production:

1. Create a small test segment (100-1000 profiles)
2. Run the incremental activation workflow
3. Manually verify the delta results match expectations
4. Gradually increase segment size

Review Workflow Execution Logs

1. Navigate to TD Data Workbench > Workflows
2. Open your Incremental Activation workflow
3. Click on a recent execution
4. Review the logs for error messages and delta record counts

Getting Help

If you continue to experience issues:

1. Collect diagnostic information:

   • Workflow configuration (sanitized, without credentials)
   • Error messages from workflow logs
   • Expected vs actual behavior
   • TD site location (US, EU, AP, JP)

2. Contact Support:

   • Submit a support ticket through the TD Console
   • Contact your Customer Success Representative
   • Include all diagnostic information collected

3. Beta Feedback:

   • As this is a Beta feature, your feedback helps improve the product
   • Report any unexpected behavior or feature requests to your CSR

Next Steps

• Incremental Activation Overview
• Configure Incremental Activation
• Incremental Activation Parameters
• Incremental Activation Mappings

Related Topics

• Activation Actions
• Activation Actions Parameters
• Creating and Editing a Workflow