Treasure Data allows the import of your data from Jira.

Prerequisites

  • Basic knowledge of Treasure Data.

Deprecation notice from Jira

Atlassian Corporation announced changes to Jira. Effective December 2018, the basic authentication method that uses passwords and cookie-based authentication to access their APIs, is deprecated. Update existing authentication as quickly as possible. Use the email and API key method when creating your authentication.

How to obtain Jira API key from your Jira web application

Login to your Jira web application server. The following example uses Jira cloud (https://id.atlassian.com)

Go to https://id.atlassian.com/manage-profile/email and note your email address. Users with basic authentication should replace the user name value with the email address.

Select Security then select Create or manage API tokens (or you can go directly to the link (https://id.atlassian.com/manage/api-tokens).


Next, select Create API token.

In the pane that appears, enter a name for your API Token and select Create.

Then on the next pane, select View to see your new API keys.

You can now see your API key. Copy it and use it to authenticate the TD Console to your Jira server.

How to transfer your Jira data to Treasure Data using the TD Console

Go to Integration Hub -> Catalog and select the JIRA tile.

Enter Email, API Key and JIRA API endpoint

Select Continue and enter a name for your new connection.

Select Done.

Go back to Integrations Hub > Authentications then select your created authentication and select New Source.

Enter a valid JQL in the New Source pop up then select Next.

When you see the preview, you can select Advanced Settings if you want to remove, add, or modify the columns which the plugin will attempt to ingest. Select Next.

In the Transfer to tab, enter the database name and table name that you want to insert the data into.

If you are creating a new database, select Create new database and give your database a name. Complete the same steps for Create new table.

Select whether to append records to an existing table or replace your existing table.

If you want to set a different partition key seed rather than use the default key, you can specify one using the popup menu.

In the Schedule tab, you can specify a one-time transfer, or you can schedule an automated recurring transfer.

Select Next. You can, optionally, rename the source. Select Done and your job is scheduled. You will be notified whenever it is done.

How to transfer your Jira data to Treasure Data using TD Toolbelt

Install ‘td’ command v0.11.9 or later

Install the newest TD Toolbelt.

$ td --version
0.11.10

Create Seed Config File (seed.yml)

Prepare seed.yml as shown in the following example, with your JIRA email with API key, JIRA URI and, by default, an empty jql query. (Username and password is not longer supported for authentication). You’ll use “replace” mode:

in:
  type: jira
  username: user@example.com
  password: APIkeysXXXXXX
  uri: https://your-domain.atlassian.net
  jql: summary ~ fix and project = 'PLT'  # example query. be sure to use singlequotes
out:
  mode: replace

The Data Connector for Jira imports all tickets that match the specified JQL.

For more details on available out modes, see Appendix.

Guess Fields (Generate load.yml)

Use connector:guess. This command automatically reads the target data, and intelligently guesses the data format.

$ td connector:guess seed.yml -o load.yml

If you open up load.yml, you’ll see guessed file format definitions including, in some cases, file formats, encodings, column names, and types.

in:
  type: jira
  username: user@example.com  password: APIkeysXXXXXX
  uri: https://your-domain.atlassian.net
  jql: ''
  columns:
  - name: id
    type: long
  - name: key
    type: string
  - name: issuetype.name
    type: string
  ...
out:
  mode: replace

Then you can preview how the system will parse the file by using preview command.

$ td connector:preview load.yml

If the system detects your column name or type unexpectedly, modify load.yml directly and preview again.

Currently, the Data Connector supports parsing of "boolean", "long", "double", "string", and "timestamp" types.

Execute Load Job

Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data is stored.

$ td connector:issue load.yml --database td_sample_db --table td_sample_table

The preceding command assumes you have already created database(td_sample_db) and table(td_sample_table). If the database or the table do not exist in TD, this command will not succeed, so create the database and table manually or use --auto-create-table option with td connector:issue command to auto create the database and table:

$ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column created_at --auto-create-table

You can assign Time Format column to the "Partitioning Key" by "--time-column" option.

Scheduled execution

You can schedule a periodic Data Connector execution for periodic JIRA import. We manage our scheduler carefully to ensure high availability. By using this feature, you no longer need a cron daemon on your local data center.

For the scheduled import, Data Connector for JIRA imports all files that match the specified JQL.

Create the schedule

A new schedule can be created by using the td connector:create command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required.

$ td connector:create \
    daily_jira_import \
    "10 0 * * *" \
    td_sample_db \
    td_sample_table \
    load.yml

The `cron` parameter also accepts three special options: `@hourly`, `@daily` and `@monthly`.

By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and can lead to unexpected schedules.

List the Schedules

You can see the list of currently scheduled entries by td connector:list.

$ td connector:list
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+
| Name              | Cron         | Timezone | Delay | Database     | Table           | Config                 |
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+
| daily_jira_import | 10 0 * * *   | UTC      | 0     | td_sample_db | td_sample_table | {"type"=>"jira", ... } |
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+

Show the Setting and History of Schedules

td connector:show shows the execution setting of a schedule entry.

% td connector:show daily_jira_import
Name     : daily_jira_import
Cron     : 10 0 * * *
Timezone : UTC
Delay    : 0
Database : td_sample_db
Table    : td_sample_table

td connector:history shows the execution history of a schedule entry. To investigate the results of each individual execution, use td job <jobid>.

% td connector:history daily_jira_import
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-18 00:10:05 +0000 | 160      |
| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-17 00:10:07 +0000 | 161      |
| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-16 00:10:03 +0000 | 152      |
| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-15 00:10:04 +0000 | 163      |
| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-14 00:10:04 +0000 | 164      |
| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-13 00:10:04 +0000 | 155      |
| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-12 00:10:05 +0000 | 156      |
| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-11 00:10:04 +0000 | 157      |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
8 rows in set

Delete the Schedule

td connector:delete removes the schedule.

$ td connector:delete daily_jira_import

Appendix

Modes for out plugin

You can specify file import mode in out section of seed.yml.

append (default)

This is the default mode and records are appended to the target table.

in:
  ...
out:
  mode: append

replace (In td 0.11.10 and later)

This mode replaces data in the target table. Any manual schema changes made to the target table remain intact with this mode.

in:
  ...
out:
  mode: replace
  • No labels