# Replacing Data Tank 1.0 With 2.0 This article covers changes around integrations with replacing Data Tank with Data Tank 2.0. As part of its commitment to security and protection of sensitive data, Treasure Data recommends reviewing the entire document before beginning any steps. This topic contains: * [Features differences between Data Tank and Data Tank 2.0](#features-differences-between-data-tank-and-data-tank-2.0) * [Change Integration with Data Tank to that with Data Tank 2.0](#change-integration-with-data-tank-to-that-with-data-tank-2.0) * [Authentications for Data Tank](#authentications-for-data-tank) * [Authentications for Data Tank 2.0](#authentications-for-data-tank-2.0) * [Import from Data Tank](#import-from-data-tank) * [Sources](#sources) * [CLI (TD Toolbelt)](#cli-td-toolbelt) * [Export Result of Hive/Trino to Data Tank](#export-result-of-hive/trino-to-data-tank) * [Query Editor on TD Console](#query-editor-on-td-console) * [td_run>: operator of Workflow](#td_run-operator-of-workflow) * [td>: operator of Workflow](#td-operator-of-workflow) * [CLI (TD Toolbelt) for Export](#cli-td-toolbelt-for-export) * [Connect to Data Tank via "pg>:" operator of Workflow](#connect-to-data-tank-via-pg-operator-of-workflow) * [Connect to Data Tanks via 3rd party tool](#connect-to-data-tanks-via-3rd-party-tool) * [References](#references) ## Features differences between Data Tank and Data Tank 2.0 In this section, feature gaps between Data Tank and Data Tank 2.0 are described. | Feature | Status | Data Tank 1.0 | Data Tank 2.0 | | --- | --- | --- | --- | | Endpoint | DIFF | Static IP address | Writer endpoint URI; associated IP address cannot be fixed | | Database name | DIFF | `datatank` | `aciddb` | | Access controls (roles and schemas) | DIFF | Treasure Data provides `tank_user` and `tank_integration_user` accounts | Users and schemas are created in **Management UI** (accounts are called **Access Keys**); access control is managed within the UI ([ref](https://docs.google.com/document/d/12Lw1bfvoXkzehSG8i02AIWC0IF_i2oNgwyP_vgzUW1g/edit#heading=h.re7k52drl4sv)) | | DDL statements | DIFF | No additional command required | `SET ROLE _owner_aciddb_` must be executed before running DDL | | SSL encryption | DIFF | Not enforced | All connections require SSL | | Maintenance windows | DIFF | No regular window; support coordinates as needed | Customers define a weekly maintenance window; patches are applied during that time (see [Understanding Data Tank 2.0 Maintenance](/products/control-panel/datatank/understanding-data-tank-2-0-maintenance)) | | IP allowlist size | DIFF | No limit | Up to 200 IP addresses | | Engine behavior gaps | DIFF | Based on PostgreSQL 12 | Based on Aurora PostgreSQL 12.13; some engine differences apply | | Custom port number | UNSUPPORTED | Default 5432, customizable via Support | Fixed to port 5432 | | Port forwarding | UNSUPPORTED | Support can add ports 80, 443, or 5439 | Not supported | | IPv6 allowlist | UNSUPPORTED | Supports IPv6 | IPv6 is not supported | | Extensions | UNSUPPORTED | `plpgsql`, `pgcrypto`, `cstore_fdw`, `mysql_fdw`, `tds_fdw`, `foreign_table_exposer` installed by default | Only `plpgsql` and `pgcrypto` available; `SERVER`, `FOREIGN DATA WRAPPER`, `FOREIGN TABLE`, and `USER MAPPING` objects cannot be migrated | | Management UI | NEW | Not available | Web interface for managing configurations, roles, schemas, and monitoring | | Log collection | NEW | Not available | PostgreSQL logs viewable in **Management UI** | | Monitoring | NEW | Dashboards provided via DataDog link | Dashboards available directly in **Management UI** | Refer to the [AWS Aurora PostgreSQL extension documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/AuroraPostgreSQL.Extensions.md#AuroraPostgreSQL.Extensions.12) for the full list of extensions supported in Data Tank 2.0 (Aurora PostgreSQL 12.13). ![](/assets/migration-from-data-tank-1-0-to-2-0-2024-04-16.bade5080991c01a24b3a336144f24f449f1a256aafadc04a7ab7ed4038ac9e21.f97e5221.png) Within the Management UI you can configure database settings, create and manage schemas and access keys, and review logs and dashboards from a single interface. | Legacy Monitoring | Management UI Monitoring | | --- | --- | | ![](/assets/datadogmonitoring.aa8164483e488c4971542e235e70da5215549003e3099c9e97f2a76af31c11d0.f97e5221.png) | ![](/assets/migration-from-data-tank-1-0-to-2-0-2024-04-16-2.4f0928af37c1403009a75ba08db9bf6a8f55ba8085a6b3baa22bff33034a0229.f97e5221.png) | ## Change Integration with Data Tank to that with Data Tank 2.0 The following two functions can be used to import records in Data Tank or export the result of Hive/Trino query to Data Tank. * [Import from PostgreSQL](https://docs.treasuredata.com/articles/int/postgresql-import-integration) * [Export to PostgreSQL](https://docs.treasuredata.com/articles/int/postgresql-export-integration) To use these integrations, you need to create an Authentication in advance in most cases. ## Authentications for Data Tank The following **Authentications** are prepared by default. `tank_integration_user`is used as a database user for both of the following **Authentications**. * datatank * datatank_cstore ![](/assets/migration-from-data-tank-1-0-to-2-0-2024-04-10-2.fcbf369e85e3caa08ebc1749ba47fdc7132c321122db485c9add5d9ac44da233.f97e5221.png) You can also create user-defined Authentications by yourself. ### Authentications for Data Tank 2.0 Authentications are not prepared by default. You need to create new Authentications by yourself. ### Import from Data Tank To import from Data Tank, you need to run a Bulk Load job. To run this type of job, you use either of the following. If you want to replace Data Tank with Data Tank 2.0, you must change the import source from Data Tank to Data Tank 2.0. * Sources(TD Console) * CLI(Toolbelt, td_load>: operator of Workflow) ### Sources If you have existing **Sources** for Data Tank, you can see those Sources connected to the Authentication in the TD Console as follows. The red arrow in the image points to the number of Sources associated with the Authentication named *databank*. ![](/assets/migration-from-data-tank-1-0-to-2-0-2024-04-10-1.12586b5e3f7b3dc023279740ce5d4eace1b20a002d1c174db96dfe12901e5003.f97e5221.png) If an existing Source imports from Data Tank 2.0, a new Source associated with the Authentication for Data Tank 2.0 is needed because each Source's authentication cannot be replaced. When you create a new Source that imports from Data Tank 2.0, it is recommended that the existing Source not be scheduled to run. ### CLI (TD Toolbelt) If you use CLI to import to Plazma DB using an Authentication, you use `td_authentication_id` option in your yaml file as follows. The Authentication ID set in this option must be changed to change the import source. In addition, you also need to change other options like `database` option. ```yaml in: type: postgresql td_authentication_id: ... ``` If you don’t use `td_authentication_id option` in the yaml file, the credentials of Data Tank would be specified instead of Authentication as shown in the following example. If so, please replace them with one of Data Tank 2.0. ```yaml in: type: postgresql host: xx.xxx.xxx.xxx user: tank_integration_user password: xxxxx ... ``` ## Export Result of Hive/Trino to Data Tank Either of the following features can exports the result of Hive/Trino job to Data Tank. If you are using these features to export to Data Tank, you will need to change the destination to export to Data Tank 2.0. * Query Editor on TD Console * td_run>: operator of Workflow * td>: operator of Workflow * CLI(Toolbelt) * REST API ### Query Editor on TD Console You can configure the export settings by selecting `**Export Results**` as shown below. Then, you need to configure it again with a new **Authentication** to replace the existing **One**. Once you have deleted the existing export settings, please use **Authentication** for Data Tank 2.0 to make new export settings. ![](/assets/migration-from-data-tank-1-0-to-2-0-2024-04-16-3.2ed05744854cbe023aa3ae20f4ba2eac2eb286bd5efc31e5d20f93707fe2cd6e.f97e5221.png) If DDL is required, you need to set `_owner__` to `Set Role` option as follows ![](/assets/setroleaciddb.9b26bc3e92b291ebfdf78fa6fb46bb8a61d22a3888be9561a7403997d55e6266.f97e5221.png) ### td_run>: operator of Workflow This workflow operator calls the existing Saved Query. Therefore, you need to do the same operation in the case of Query Editor. ### td>: operator of Workflow For this Workflow operator, you specify your **Authentication** name for `result_connection` parameter as follows. Replace this parameter with the **Authentication** for Data Tank 2.0 as well as any other parameters. ```yaml +export_task: td>: queries/sample.sql result_connection: data_tank # Authentication Name result_settings: database: datatank table: ... ... ``` In addition, if DDL is required, you need to specify`set_role` option as follows. ```yaml +export_task: td>: queries/sample.sql result_connection: result_settings: database: aciddb table: ... schema: set_role: _owner_aciddb_ ... ``` If you don’t specify your Authentication for `result_connection`, use `result_url`parameter instead of `result_connection` and `result_settings`as shown. ```yaml +export_task: td>: queries/sample.sql result_url: postgresql://tank_integration_user:xxx@...... ``` In this case, please modify the value for `result_url` parameter. In addition, add `set_role=_owner_aciddb_` as query parameter if DDL statement is required. ### CLI (Toolbelt) for Export For CLI, you specify the information for export in URL format. For example, you can export the result of the query to Data Tank by using `td query` command as follows. As you specify the information for export to `-r` option directly, please specify the setting for Data Tank 2.0. ```bash $ td query -d kazzy_test -w -r "postgresql://tank_integration_user:@/datatank/?mode=replace" "SELECT 1 AS col1;" ``` ## Connect to Data Tank via "pg>:" operator of Workflow You can run any query statement in Data Tank from Workflow by using `pg>:` operator as shown in the following example. ```yaml _export: pg: host: .... port: 5432 database: datatank user: tank_integration_user schema: public +access_data_tank: pg>: queries/sample.sql ... ``` You need to change the values set for the options of pg>: operator in order to run the query statements in Data Tank 2.0. If you use DDL, you need to add `SET ROLE` statement before the DDL statements. ## Connect to Data Tanks via 3rd party tool To connect to your Data Tank via any third-party tool, you specify the connection information in the tool. Replace them with one of Data Tank 2.0. ## References * [About Data Tank 2.0](/products/control-panel/datatank/about-data-tank-2-0) * [Integrating Data Tank 2.0 with Plazma DB](/products/control-panel/datatank/about-integrating-data-tank-2-0-with-plazma-db) * [Understanding Data Tank 2.0 Maintenance](/products/control-panel/datatank/understanding-data-tank-2-0-maintenance) * [Managing Data Tank 2.0 with the Management UI](/products/control-panel/datatank/managing-data-tank-2-0-with-the-management-ui)