Page tree
Skip to end of metadata
Go to start of metadata

Use this page as a reference for the primary operators that you can use in Treasure Workflow.

TD Workflow is based on the Treasure Data created OSS project Digdag. Most, but not all, Digdag operators can be used as part of Treasure Workflow.

Network Operator

Description

mail>:

Sending email

http>:

Making HTTP requests

mail>:

mail> operator sends an email.

To use Gmail SMTP server, you need to:

  • Generate a new app password at App passwords. This requires enabling 2-Step Verification first.

  • Enable access for less secure apps at Less secure apps. This works even if 2-Step Verification is not enabled.

    _export:
      mail:
        from: "you@gmail.com"
    
    +step1:
      mail>: body.txt
      subject: workflow started
      to: [me@example.com]
      cc: [foo@example.com,bar@example.com]
    
    +step2:
      mail>:
        data: this is email body embedded in a .dig file
      subject: workflow started
      to: [me@example.com]
      bcc: [foo@example.com,bar@example.com]
    
    +step3:
      sh>: this_task_might_fail.sh
      _error:
        mail>: body.txt
        subject: this workflow failed
        to: [me@example.com]
    

Secrets

  • mail.host: HOST

    SMTP host name.

    Examples:



    mail.host: smtp.gmail.com



  • mail.port: PORT

    SMTP port number.

    Examples:



    mail.port: 587



  • mail.username: NAME

    SMTP login username.

    Examples:



    mail.username: me



  • mail.password: PASSWORD

    SMTP login password.

    Examples:



    mail.password: MyPaSsWoRd



    If you encountered the Unsecure 'password' parameter is deprecated. message, confirm that you used the following command:

# Server Mode
digdag secret --project test_mail --set mail.password=xxxxx

# Local Mode stores the parameter into '~.config/digdag/secrets/mail.password'
digdag secret --local --set mail.password=xxxxx
  • mail.tls: BOOLEAN Enables TLS handshake.

    Examples:

    mail.tls: true

  • mail.ssl: BOOLEAN

    Enables legacy SSL encryption.

    Examples:

    mail.ssl: false

Options

  • mail>: FILE

    Path to a mail body template file. This file can contain ${...} syntax to embed variables. Alternatively, you can set {data: TEXT} to embed body text in the .dig file.

    Examples:

    mail>: mail_body.txt

    • or :command:mail>: {data: "Hello, this is from Digdag"}
  • subject: SUBJECT

    Subject of the email.

    Examples:

    subject: Mail From Digdag

  • to: [ADDR1, ADDR2, …]

    To addresses.

    Examples:

    to: [analyst@examile.com]

  • from: ADDR From address.

    Examples:

    from: admin@example.com

  • host: NAME

    SMTP host name.

    Examples:

    host: smtp.gmail.com

  • port: NAME

    SMTP port number.

    Examples:

    port: 587

  • username: NAME

    SMTP login username.

    Examples:

    username: me

  • tls: BOOLEAN

    Enables TLS handshake.

    Examples:

    tls: true

  • ssl: BOOLEAN

    Enables legacy SSL encryption.

    Examples:

    ssl: false

  • html: BOOLEAN

    Uses HTML mail (default: false).

    Examples:

    html: true

  • debug: BOOLEAN

    Shows debug logs (default: false).

    Examples:

    debug: false

  • attach_files: ARRAY

    Attach files. Each element is an object of:

    • path: FILE: Path to a file to attach.

    • content_type: STRING: Content-Type of this file. Default is application/octet-stream.

    • filename: NAME: Name of this file. Default is base name of the path.

    Example:

    attach_files:
      - path: data.csv
      - path: output.dat
        filename: workflow_result_data.csv
      - path: images/image1.png
        content_type: image/png


http>:

http> operator can be used to make HTTP requests.

+fetch: 
  http>: https://api.example.com/foobars 
  store_content: true

+process: 
  for_each>:
  foobar: ${http.last_content}
  _do:
  bq>: query.sql

+notify: 
  http>: https://api.example.com/data/sessions/{$session_uuid} 
  method: POST 
  content:
    status: RUNNING
    time: ${session_time}

Secrets

  • http.authorization: STRING

    A string that should be included in the HTTP request as the value of the Authorizationheader. This can be used to authenticate using e.g. Oauth bearer tokens.

  • http.user: STRING

    A user that should be used to authenticate using Basic Authentication.

  • http.password: STRING

    A password that should be used to authenticate using Basic Authentication.

  • http.uri: URI

    The URI of the HTTP request. This can be used instead of putting the URI on the operator command line in case the URI contains sensitive information.

Parameters

  • http>: URI

    The URI of the HTTP request.

    Examples:

    http>: https://api.example.com/foobar

    http>: https://api.example.com/data/sessions/{$session_uuid}

  • method: STRING

    The method of the HTTP request. Default: GET.

    Examples:

    method: POST

    method: DELETE

  • content: STRING | INTEGER | BOOLEAN | OBJECT | ARRAY The content of the HTTP request. Default: No content.

    Scalars (i.e. strings, integers, booleans, etc) will by default be sent as plain text. Objects and arrays by default are JSON serialized. The content_format parameter can be used to control the content serialization format.

    content: 'hello world'

    content: '${session_time}'

    content: status: RUNNING time: ${session_time}

  • content_format: text | json | form

    The serialization format of the content of the HTTP request. Default: Inferred from the contentparameter value type. Objects and arrays use json by default. Other value types default to text.

    • text: Send raw content as Content-Type: text/plain. This requires that the content parameter is not array or an object.

    • json: Serialize the content as JSON and send it as Content-Type: application/json. This format can handle any content parameter value type.

    • form: Encode content as an HTML form and send it as Content-Type: application/x-www-form-urlencoded. This requires the content parameter value to be an object.

    content: 'hello world @ ${session_time}' content_format: text

    content: status: RUNNING time: ${session_time} content_format: json

    content: status: RUNNING time: ${session_time} content_format: form

  • content_type: STRING

    Override the inferred Content-Type header.

    content: | <?xml version=“1.0” encoding=“UTF-8”?>


    <status>RUNNING</status>
    <time>${session_time}</time>

    content_format: text content_type: application/xml ```

  • store_content: BOOLEAN

    Whether to store the content of the response. Default: false.

  • headers: LIST OF KEY-VALUE PAIRS

    Additional custom headers to send with the HTTP request.

    headers:


    - Accept: application/json
    - X-Foo: bar
    - Baz: quux
  • retry: BOOLEAN

    Whether to retry ephemeral errors. Default: true if the request method is GET, HEAD, OPTIONSor TRACE. Otherwise false.

    Client 4xx errors (except for 408 Request Timeout and 429 Too Many Requests) will not be retried even if retry is set to true.

    Enabling retries might cause the target endpoint to receive multiple duplicate HTTP requests. Thus retries should only be enabled if duplicated requests are tolerable. For example, when the outcome of the HTTP request is idempotent. Idempotent values can be applied multiple times without changing the result beyond the initial application.

  • No labels