Skip to content
Products
/
/
/
CD with CircleCI
Last updated

Continuous Deployment of Workflows With CircleCI

TD Workflows can be used with version control tools like Git and continuous integration tools such as CircleCI. In this tutorial, we’ll go through how to set up a Continuous Deployment pipeline using GitHub and CircleCI.

Prerequisites

GitHub repository

If you have not already done so, create a git repository on GitHub with your workflow project. For more information on how to use Git and GitHub, see the GitHub Bootcamp.

Configure CircleCI

Add a CircleCI configuration file to the project. The file should be named config.yml, be in the top directory called .circleci, and have the following contents. Then commit and push to GitHub.

version: 2.1
executors:
  my_executor:
    docker:
      - image: ruby:2.7.2-buster

jobs:
  digdag:
    executor: my_executor
    steps:
      - checkout
      - run:
          name: Setup
          command: |
            gem install td
            yes | td -e https://api.treasuredata.com -k dummy wf || true
      - run:
          name: Check config
          command: cd workflow && td wf check
      - run:
          name: Push project
          command: td -e https://api.treasuredata.com -k $TD_API_KEY wf push $WF_PROJECT -r `date -u +"%Y-%m-%dT%H:%M:%SZ"`-`git rev-parse HEAD`

workflows:
  version: 2
  build-n-deploy:
    jobs:
      - digdag
        # If you would like to restrict branches/tags to running Digdag jobs, use `filters` option.
        # Following filter means `digdag` job will run only `dev` and `user-` starting branches.
        # @see: https://circleci.com/docs/2.0/workflows/
        #  - digdag:
        #      filters:
        #        branches:
        #          only:
        #            - dev
        #            - /user-.*/

The configuration specifies that CircleCI should deploy changes on the masterbranch only. This ensures that changes on other branches and pull requests are not deployed until they are merged to master.

For more information on CircleCI configuration, see the CircleCI configuration documentation.

Connect CircleCI

  1. Open the CircleCI dashboard.

  2. Select Add Projects on the left-hand side.

  3. Select your account. The GitHub repository containing your TD Workflow project should be listed.

  4. Select Build Project on the right-hand side to start the continuous deployment pipeline.

CircleCI performs the first run of the pipeline, and will fail because we have not yet configured TD credentials.

Configure TD Credentials

For workflows to be pushed to Treasure Data, you must configure a TD API Key for the project.

  1. Open the CircleCI dashboard again. The workflow project should be listed on the left.

  2. Select the workflow project.

  3. Select Project Settings in the upper right corner.

  4. Select Environment Variables under Build Settings on the left-hand side.

  5. Type TD_API_KEY into the Name field and your TD API key in the Value field.

  6. Select Save variables.

Start Deploying

  1. Open the CircleCI dashboard again and select the failed workflow project build.

  2. Select Rebuild in the upper right corner.

CircleCI successfully pushes your workflow to Treasure Data and does this every time you push a change to GitHub.

Troubleshooting

If you see the following error, add https:// to the endpoint.

error: RESTEASY004655: Unable to invoke request (processing)
Previous page
Next page