Visit our new documentation site! This documentation page is no longer updated.

Workflows Secrets Management

Table of Contents

Introduction

In this tutorial you will learn how to configure a workflow to use specific TD apikeys and PostgreSQL credentials.

If you have not already done so, start by going through the TD Workflows introductory tutorial.

Background

As TD apikeys, database credentials etc can be used to access your potentially sensitive business data, it is important to treat them securely. For this purpose, TD Workflows offers a secure secret management system that can be used to securely manage credentials separately from normal workflow parameters.

TD Apikeys

When a workflow is pushed to and runs on the TD Workflows service, it automatically uses the TD apikey associated with the TD account that was specified when the TD Toolbelt was configured using td account. To use another apikey, uploaded it to the TD Workflows service using the td wf secrets command. In the below tutorial we will go through configuring a workflow to use a specific TD apikey.

Tutorial

Upload the apikey to use by running the below command and enter your apikey when prompted. The apikey will not be visible in the terminal. Finishing by pressing enter.

$ td wf secrets --project nasdaq_analysis --set td.apikey

Now all workflows in this project will use this apikey. Try it out by starting the workflow.

$ td wf start nasdaq_analysis nasdaq_analysis --session now

The apikey specified will not show up in the workflow logs, but the workflow should run successfully. If an invalid apikey was specified the run will fail.

To change the apikey, run the secrets set command again and specify another apikey.

To revert to using the default apikey, delete the uploaded apikey.

$ td wf secrets --project nasdaq_analysis --delete td.apikey

Multiple apikeys

Sometimes it is desirable to configure tasks in a workflow to use different apikeys in order to access different TD accounts.

First upload two different apikeys.

$ td wf secrets --project nasdaq_analysis --set apikey1

$ td wf secrets --project nasdaq_analysis --set apikey2

Now let’s configure the two tasks to use the two different apikeys.

$ cat > nasdaq_analysis.dig <<EOF
timezone: UTC

schedule:
  daily>: 07:00:00

_export:
  td:
    database: workflow_temp

+task1:
  _secrets:
    td:
      apikey: apikey1
  td>: queries/daily_open.sql
  create_table: daily_open

+task2:
  _secrets:
    td:
      apikey: apikey2
  td>: queries/monthly_open.sql
  create_table: monthly_open
EOF

Start the workflow and check that it runs successfully.

$ td wf start nasdaq_analysis nasdaq_analysis --session now
Untitled-3
For the workflow above to successfully run, you might need to create the database workflow_temp in the accounts if the two respective apikeys, if it does not already exist.

PostgreSQL credentials

This section of the tutorial assumes that you already have a workflow using the pg> operator.

To securely configure the PostgreSQL user and password, upload them to TD Workflows using the td wf secrets command.

$ td wf secrets --project my_project --set pg.user

$ td wf secrets --project my_project --set pg.password

Also make sure to remove pg.user and pg.password from the workflow file, if present.

Now all workflows in this project will use the specified user and password when executing pg> operators. Several other operator configuration options can also be securely specified if desired.

  • host
  • port
  • user
  • password
  • database
  • ssl
  • connect_timeout
  • socket_timeout
  • schema

For more information about the different configuration options please see the Digdag documentation.

To use different sets of PostgreSQL credentials within a workflow project, upload them with different names.

$ td wf secrets --project my_project --set db1.pg.user

$ td wf secrets --project my_project --set db1.pg.password

$ td wf secrets --project my_project --set db2.pg.user

$ td wf secrets --project my_project --set db2.pg.password

Then refer to the credentials using the db1 and db2 names, respectively. (Note that the names can be freely chosen).

+task1:
  _secrets:
    pg: db1
  pg>: query1.sql

+task2:
  _secrets:
    pg: db2
  pg>: query2.sql

Now task1 will use the db1 credentials and task2 will use the db2 credentials.

Uploading secrets from a file

When managing several sets of credentials, it can be more convenient to upload them all in one go using a file.

# Create a file with credential key-value pairs.
$ cat > credentials.yml <<EOF
db1.pg.user: user1
db1.pg.password: pw1
db2.pg.user: user2
db2.pg.password: pw2
db3.pg.user: user3
db3.pg.password: pw3
EOF

# Upload the credentials to TD Workflows
td wf secrets --project my_project --set @credentials.yml

Listing secrets

To list the secrets that have been uploaded to a project, omit the --set option when running the secrets command.

$ td wf secrets --project my_project

Local secrets

To use the secrets on local mode, omit the --local option instead of --project when running the secrets command.

$ td wf secrets --local --set @credentials.yml

Deleting secrets

To delete credentials that have been uploaded to a project, using the --delete option and specify one or more secrets to delete.

$ td wf secrets --project my_project --delete db2.pg.user db2.pg.password

Downloading secrets

For security reasons, it is not possible to download secrets that have been uploaded to the TD Workflows service.


Last modified: Mar 07 2018 12:36:56 UTC

If this article is incorrect or outdated, or omits critical information, let us know. For all other issues, access our support channels.