# Jira Import Integration

Treasure Data allows the import of your data from Jira.

# Prerequisites

- Basic knowledge of Treasure Data.


# Deprecation notice from Jira

Atlassian Corporation announced changes to Jira. Effective December 2018, [the basic authentication method that uses passwords and cookie-based authentication to access their APIs, is deprecated](https://developer.atlassian.com/cloud/jira/platform/deprecation-notice-basic-auth-and-cookie-based-auth/). Update existing authentication as quickly as possible. **Use the email and API key method when creating your authentication.**

# How to obtain Jira API key from your Jira web application

Login to your Jira web application server. The following example uses Jira cloud ([https://id.atlassian.com](https://id.atlassian.com/))

Go to [https://id.atlassian.com/manage-profile/email](https://id.atlassian.com/manage-profile/email) and note your email address. Users with basic authentication should **replace the user name value with the email address**.

![](/assets/image-20190923-174923.f5fd72ea954b8d15bbb32e810dfa86632a9e64e0a0b5e1026ecaec8c13aa50c9.7eaf6023.png)

Select **Security** then select **Create or manage API tokens** (or you can go directly to the link ([https://id.atlassian.com/manage/api-tokens](https://id.atlassian.com/manage/api-tokens)).

![](/assets/image-20190923-175000.3b01217ca7bb43a288f5ba40fb6683f4ba10bb399ad3aa1ff6bf30ce421f2838.7eaf6023.png)

Next, select **Create API token**.

![](/assets/image-20190923-175059.609550313e8eed1419fddb1fd92c370ee8dfcbf086a4042bb26366f1c9743e24.7eaf6023.png)

In the pane that appears, enter a name for your API Token and select **Create**.

![](/assets/image-20190923-175128.7cd7943b287141b8820f10b40b2e3230ea3c5177148becb651a3c97aa0d5b031.7eaf6023.png)

Then on the next pane, select **View** to see your new API keys.

![](/assets/image-20190923-175150.8553c49d68938935313d8a23c78d7440964b8f9994068736f6b0f6915f3718c5.7eaf6023.png)

You can now see your API key. Copy it and use it to authenticate the TD Console to your Jira server.

![](/assets/image-20190923-175212.eb533c011235c3cb53072e02166ab6b0f1cca0e234cae7539370fc6d2845dee7.7eaf6023.png)

# How to transfer your Jira data to Treasure Data using the TD Console

Go to **Integration Hub -> Catalog** and select the **JIRA** tile.

![](/assets/image-20190923-175249.036594bd3f360c1b2134dcb0d6e267aa5e93c9dda5ee3ecacd0a2f40813505ed.7eaf6023.png)

Enter **Email**, **API Key** and **JIRA API endpoint**

![](/assets/image-20190923-175328.5afef6e2cba87dcc04c0179890476743afc21ca261c557785937e207fd833590.7eaf6023.png)

Select **Continue** and enter a name for your new connection.

![](/assets/image-20190923-175409.a6d53732c4959e11d2dfe6db7ce72667b39fc48fd5978abd90cb98d5da73fc91.7eaf6023.png)

Select **Done**.

Go back to **Integrations Hub > Authentications** then select your created authentication and select **New Source**.

![](/assets/image-20190923-175444.417352d7ba29572eacadab91ef6ac3764ac379eca0a6d1360dc473f24bcfd55f.7eaf6023.png)

Enter a valid JQL in the **New Source** pop up then select **Next.**

![](/assets/image-20190923-175519.6cb6f3bba87c77adfcf0d2b1d394e6115cfa207e08d19223203b62f0674f4992.7eaf6023.png)

When you see the preview, you can select **Advanced Settings** if you want to remove, add, or modify the columns which the plugin will attempt to ingest. Select **Next.**

![](/assets/image-20190923-175548.ae8fc13e1de315fb34a16134e13192a02b60e781fa03fa195ed4fe7144dca384.7eaf6023.png)

In the **Transfer to** tab, enter the database name and table name that you want to insert the data into.

![](/assets/image-20190923-175622.b64aaa7e08f5bf3dfa01a244c3d7b2226a08689c61767c611d4c882f23f75696.7eaf6023.png)

If you are creating a new database, select **Create new database** and give your database a name. Complete the same steps for **Create new table.**

Select whether to **append** records to an existing table or **replace** your existing table.

If you want to set a different **partition key seed** rather than use the default key, you can specify one using the popup menu.

![](/assets/image-20190923-175654.f981e307b7042086416363bd43db4912444239ced64586ebcb19984e4b15cda8.7eaf6023.png)

In the **Schedule** tab, you can specify a one-time transfer, or you can schedule an automated recurring transfer.

![](/assets/image-20190923-175732.7012f0c64834a6f0c798b5df04bf935fefe63e8b591f900ea48f5273d67394fa.7eaf6023.png)

Select **Next**. You can, optionally, rename the source. Select **Done** andyour job is scheduled. You will be notified whenever it is done.

![](/assets/image-20190923-175757.43374ab2c2453418ca1ea6c72d33ba57aa6a5864741c6691e7aacd0c8c5b9314.7eaf6023.png)

# How to transfer your Jira data to Treasure Data using TD Toolbelt

## Install ‘td’ command v0.11.9 or later

Install the newest [TD Toolbelt](https://toolbelt.treasuredata.com/).


```
$ td --version
0.11.10
```

## Create Seed Config File (seed.yml)

Prepare `seed.yml` as shown in the following example, with your JIRA email with API key, JIRA URI and, by default, an empty jql query. (Username and password is not longer supported for authentication). You’ll use “replace” mode:


```
in:
  type: jira
  username: user@example.com
  password: APIkeysXXXXXX
  uri: https://your-domain.atlassian.net
  jql: summary ~ fix and project = 'PLT'  # example query. be sure to use singlequotes
out:
  mode: replace
```

The Data Connector for Jira imports all tickets that match the specified JQL.

For more details on available `out` modes, see Appendix.

## Guess Fields (Generate load.yml)

Use `connector:guess`. This command automatically reads the target data, and intelligently guesses the data format.


```
$ td connector:guess seed.yml -o load.yml
```

If you open up `load.yml`, you’ll see guessed file format definitions including, in some cases, file formats, encodings, column names, and types.


```
in:
  type: jira
  username: user@example.com  password: APIkeysXXXXXX
  uri: https://your-domain.atlassian.net
  jql: ''
  columns:
  - name: id
    type: long
  - name: key
    type: string
  - name: issuetype.name
    type: string
  ...
out:
  mode: replace
```

Then you can preview how the system will parse the file by using `preview` command.


```
$ td connector:preview load.yml
```

If the system detects your column name or type unexpectedly, modify `load.yml` directly and preview again.

Currently, the Data Connector supports parsing of "boolean", "long", "double", "string", and "timestamp" types.

## Execute Load Job

Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data is stored.


```
$ td connector:issue load.yml --database td_sample_db --table td_sample_table
```

The preceding command assumes you have already created *database(td_sample_db)* and *table(td_sample_table)*. If the database or the table do not exist in TD, this command will not succeed, so [create the database and table](https://docs.treasuredata.com/articles/project-product-documentation/database-and-table-management) manually or use `--auto-create-table` option with `td connector:issue` command to auto create the database and table:


```
$ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column created_at --auto-create-table
```

You can assign Time Format column to the "Partitioning Key" by "--time-column" option.

## Scheduled execution

You can schedule a periodic Data Connector execution for periodic JIRA import. We manage our scheduler carefully to ensure high availability. By using this feature, you no longer need a `cron` daemon on your local data center.

For the scheduled import, Data Connector for JIRA imports all files that match the specified JQL.

## Create the schedule

A new schedule can be created by using the `td connector:create` command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required.


```
$ td connector:create     daily_jira_import     "10 0 * * *"     td_sample_db     td_sample_table     load.yml
```

The `cron` parameter also accepts three special options: `@hourly`, `@daily` and `@monthly`. | By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and can lead to unexpected schedules.

## List the Schedules

You can see the list of currently scheduled entries by `td connector:list`.


```
$ td connector:list
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+
| Name              | Cron         | Timezone | Delay | Database     | Table           | Config                 |
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+
| daily_jira_import | 10 0 * * *   | UTC      | 0     | td_sample_db | td_sample_table | {"type"=>"jira", ... } |
+-------------------+--------------+----------+-------+--------------+-----------------+------------------------+
```

## Show the Setting and History of Schedules

`td connector:show` shows the execution setting of a schedule entry.


```
$ td connector:show daily_jira_import
```

`td connector:history` shows the execution history of a schedule entry. To investigate the results of each individual execution, use `td job jobid`.


```
% td connector:history daily_jira_import
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| JobID  | Status  | Records | Database     | Table           | Priority | Started                   | Duration |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
| 578066 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-18 00:10:05 +0000 | 160      |
| 577968 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-17 00:10:07 +0000 | 161      |
| 577914 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-16 00:10:03 +0000 | 152      |
| 577872 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-15 00:10:04 +0000 | 163      |
| 577810 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-14 00:10:04 +0000 | 164      |
| 577766 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-13 00:10:04 +0000 | 155      |
| 577710 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-12 00:10:05 +0000 | 156      |
| 577610 | success | 10000   | td_sample_db | td_sample_table | 0        | 2015-04-11 00:10:04 +0000 | 157      |
+--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+
8 rows in set
```

## Delete the Schedule

`td connector:delete` removes the schedule.


```
$ td connector:delete daily_jira_import
```

# Appendix

## Modes for out plugin

You can specify file import mode in `out` section of seed.yml.

### append (default)

This is the default mode and records are appended to the target table.


```
in:
  ...
out:
  mode: append
```

### replace (In td 0.11.10 and later)

This mode replaces data in the target table. Any manual schema changes made to the target table remain intact with this mode.


```
in:
  ...
out:
  mode: replace
```