Skip to content
/
Using TD Workflow With TD...
Last updated

Using TD Workflow With TD Integrations

Most batch import and export integrations can be added to workflows as part of a more advanced data pipeline. The basic steps are:

  • Prepare a workflow definition, which is a YAML file, using the td_load> operator for an import integration and the td> operator for an export integration.
  • Run the workflow on the TD Console, or from a command-line interface

Use a pre-defined Source of an import integration

If you have already created a Source for an import integration, you can run it in a workflow using its unique ID

  1. Identify the target Source on the Integration Hub

  2. Select the **...** menu, and select Copy Unique ID

  3. Define a workflow task using the td_load> operator.

    +my_first_worklfow_step:     td_load>: unique_id_of_your_source

Use a configuration file for an import integration

Alternatively, you can provide all the parameters of the import connector in a YAML file, and refer to it in the workflow definition

  1. Prepare a configuration file for your import integration, ex: load_config.yml. Examples and the full parameter explanation can be found in the document of each integration

  2. Define a workflow refers to it

    +my_first_workflow_step:  
   td_load>: config/load_config.yml  
   database: my_database_name  
   table: my_table_name

Define a workflow involving an export integration

There are a few ways to define a workflow task to query and upload the results to an export integration

  1. Use the td> operator and the name of a pre-defined Authentication

    +query_and_export:    
   td>: queries/sample.sql  
   database: my_database  
   result_connection: ads_with_oauth    
   result_settings:        
      parameter1: value1    
      parameter2: value2    ....

    The parameter explanation can be found on each integration document, as the one below from the SFCC Export Integration

  2. Use the td> operator and the ID of a pre-defined Authentication

    +query_data_and_write_result_with_existing_authentication_by_url:  
   td>:   query: 'select * from wf_test'  
   database: my_database  
   result_url: '{"type": "google_adwords_v2", "td_authentication_id": 330392, "bucket": "qa-auto-test-3", "path": "viet/assume_role_from_eu01.csv"}'

  3. Use the td_run> operator and a saved query

    +query_a_saved_sql:  
   td_run>: presto_select_all_s3v2  
   database: my_database  
   result_url: '{"type":"s3_v2", "td_authentication_id":260370, "bucket": "qqus-east-1", "path":"foo/wf_jan16_1444.csv"}'

Run a workflow

There are two options to define and run a workflow

  1. Use the graphical interface on TD Console > Data Workbench > Workflows
  2. Use the command-line interface of TD Toolbelt

While using the TD Console is more straightforward than the command-line interface, the TD CLI tool offers options for more complex configurations.

Refer here for more details about the syntax and usages of the TD Workflow

Previous page
Next page