Visit our new documentation site! This documentation page is no longer updated.

Continous Deployment of Workflows

Table of Contents


TD Workflows can be used with version control tools like Git and continous integration tools such as CircleCI. In this tutorial we’ll go through how to set up a Continous Deployment pipeline using GitHub and CircleCI.

This tutorial assumes that you’re already somewhat familiar with TD Workflows. If not, please check out the onboarding tutorial first.


For this tutorial, you will need:


GitHub repository

If you have not already done so, please create a git repository on GitHub with your workflow project. For more information on how to use Git and GitHub, please see the GitHub Bootcamp.

Configure CircleCI

Add a CircleCI configuration file to the project. The file should be named circle.yml, be in the top directory of the project and have the following contents. Then commit and push to GitHub.

  timezone: UTC
    version: oraclejdk8

    - mkdir -p ~/bin/
    - curl -o ~/bin/digdag -L ""
    - chmod +x ~/bin/digdag

    - digdag check

    branch: master
      - >
        digdag push $(basename $(git rev-parse --show-toplevel))
        -X client.http.headers.authorization="TD1 $TD_API_KEY"
        -r `date -u +"%Y-%m-%dT%H:%M:%SZ"`-`git rev-parse HEAD`

Note that the above configuration specifies that CircleCI should deploy changes on the master branch only. This ensures that changes on other branches and pull requests are not deployed until they are merged to master.

For more information on CircleCI configuration, please see the CircleCI configuration documentation.

Connect CircleCI

Now open the CircleCI dashboard, click “Add Projects” on the left hand side and select your account. The GitHub repository containing your TD Workflow project should be listed. Click “Build Project” on the right hand side in order to start the continous deployment pipeline.

CircleCI will now perform the first run of the pipeline, which will fail because we have not yet configured TD credentials.

Configure TD Credentials

In order for workflows to be pushed to Treasure Data, you will need to configure a TD API Key for the project.

  1. Open the CircleCI dashboard again. The workflow project should be listed on the left.
  2. Click the workflow project and then “Project Settings” in the upper right corner.
  3. Click “Environment Variables” under “Build Settings” on the left hand side.
  4. Enter TD_API_KEY into the “Name” field and your TD api key in the “Value” field.
  5. Click “Save variables”.

Start Deploying

  1. Open the CircleCI dashboard again and click the failed workflow project build.
  2. Click “Rebuild” in the upper right corner.

CircleCI should now successfully push your workflow to Treasure Data and will do this every time you push a change to GitHub.

Last modified: Aug 30 2016 09:01:36 UTC

If this article is incorrect or outdated, or omits critical information, let us know. For all other issues, access our support channels.