Workflows often need to use specific TD API keys and PostgreSQL credentials.
As TD API keys, database credentials, and other objects can be used to access your potentially sensitive business data, it is important to treat them securely.
TD Workflows offers a secure secret management system that you can use to securely manage credentials separately from normal workflow parameters.
See also Secrets Best Practices.
Review the TD Workflows SFTP Data Connector Example.
For security reasons, it is not possible to download secrets that have been uploaded to the TD Workflows service.
Default Workflow Permissions
When a workflow is pushed to TD Workflows, it automatically uses the permission of the user who pushed the workflow.
To run your workflow with another user's permission, upload the other key to the TD Workflows service using the
td wf secrets command. You configure a workflow using a specific TD API key.
Setting, Changing, and Reverting an API Key
Upload the API key to use by running the below command and enter your API key when prompted. The API key is not visible in the terminal. Then press Enter.
Now all workflows in this project will use the API key. Try it out by starting the workflow.
The API key specified is not visible in the workflow logs, but the workflow should run successfully. If an invalid API key was specified the run fails.
To change the API key, run the secrets set command again and specify another API key.
To revert to using the default API key, delete the uploaded API key.
Configuring a Workflow to Use Multiple API Keys
Sometimes you might want to configure tasks in a workflow to use different API keys to access different TD accounts.
For the workflow to run successfully, you might need to create the database workflow_temp in the accounts of the two respective API keys, if a temp file does not already exist.
Upload two different API keys.
Configure the two tasks to use the two different API keys.
Start the workflow and check that it runs successfully.
Securely Configuring PostgreSQL Credentials
This section assumes that you already have a workflow that uses the
To securely configure the PostgreSQL user and password, upload them to TD Workflows using the
td wf secrets command.
Make sure to remove
pg.password from the workflow file, if present.
All workflows in this project use the specified user and password when executing
pg>operators. Several other operator configuration options can also be securely defined, including:
For more information about the different configuration options, see the Digdag documentation.
To use different sets of PostgreSQL credentials within a workflow project, upload them with different names.
Then refer to the credentials using the
db2 names, respectively.
The names can be freely chosen.
task1 will use the
db1 credentials and
task2 will use the
Uploading Secrets from a File
When managing several sets of credentials, it can be more convenient to upload them all at one time using a file.
To list the secrets that have been uploaded to a project, omit the
--set option when running the secrets command.
To use the secrets on local mode, omit the
--local option instead of
--project when running the secrets command.
To delete credentials that have been uploaded to a project, using the
--delete option and specify one or more secrets to delete.