This is a summary of new Treasure Data features and improvements introduced in the February 1st, 2020 release. If you have any product feature requests, submit them to

Optionally, you can view a video summary of our February releases:

Workflow Utilization Dashboard

The Workflow Utilization dashboard shows your recent and historical use of workflows in Treasure Data. You can see session success rates for user-defined workflows, including custom script workflows. Use the dashboard to view the number of workflow run attempts and duration of workflow runs.  You can also monitor your resources by viewing run hours and remaining resource hours for custom scripts. 

The Workflow Utilization dashboard is available. Contact your Customer Success Representative for access information.

With the Workflow Utilization board feature enabled, navigate to the Control Panel. Click Utilization > Workflow Utilization.

Workflow Operator Updates

You can use the following operators for more efficient queries:

  • s3_wait>: This operator enables a timeout option to terminate the service waiting on data to land in the s3 bucket.

  • td_result_export>: This operator has been updated with a result_connection parameter. Results can be exported to a specific destination. 

For more information, see the Reference for Amazon Web Services Operators and the Reference for Treasure Data Operators.

Workflow Message Helps to Prevent Workflow Overwrites

When you are editing a workflow at the same time that others are editing the same workflow, you risk applying your update to an older workflow revision. In TD Console, the following message alerts you to prevent overwrite.

Improved Experience for Import Integrations

Treasure Data offers some new features that improve your experience when creating and editing data transfers for sources. Changes include the following capabilities:

  • After you  configure your data connector, you can:

    • save it and run the import immediately

    • save the configured data connector to run at a later time

  • For every new data transfer, you can select a database and table, instead of a default database and table being selected for you. This is particularly useful when you already have a similar, existing data transfer and want the specified location for the imported data to be different.

You can view all data connectors for import through a new interface, which will be viewable in TD Console before the end of February. 

Data Connectors

Google Sheet Export Re-Authentication Required

You must re-authenticate to access Google Drive metadata. Refer to Google Sheets - Export.

The current version of the Google Sheets V3 connector is scheduled to be deprecated early March 2020. Before deprecation, you must re-authenticate all your existing Google Sheet Authentications. Failure to re-authenticate results in a disruption of your Google sheets jobs. For migrating instructions, see the Appendix in Google Sheets - Export.

Twitter Audience Insights

Twitter deprecated the Twitter Ads API endpoint that supports audience insight on January 30th 2020. As a result,  the Twitter Audience Insights Integration will be deprecated 02/05/2020. 

Microsoft Azure BLOB Storage Improvement Coming Soon

The incremental loading option is available for access through TD Console. An issue that causes duplicate file ingestion has been fixed. Contact TD Support to ensure that the configuration of your jobs are up to date with the new implementation.

Read more about the import data integration for Microsoft Azure BLOB Storage.

Google Cloud Storage Improvement

The data integration for Google Cloud Storage enables the import of the contents of .tsv and .csv files stored in your GCS bucket. You can use the data connector to handle multiple byte characters, specified in the  last_path parameter, properly.

AWS Redshift Use of Serial Copy Mode

The export data integration for Redshift supports serial COPY mode. Use serial COPY mode if you encounter the following error when exporting data to Redshift:  

> Detail: Process xxxx waits for AccessExclusiveLock on relation xxxx of database xxxx; blocked by process xxxx”

Otherwise, the serial COPY mode option is not recommended because of the negative impact to performance.

Microsoft SQL Server Improvement

In Microsoft SQL Server, you can specify the schema name of the database. When using the import or export data connector, you can see the schema information in the log.

Bug Fixes and Other Improvements

Issues that now have the correct behavior in TD Console include the following:

  • When an account owner sends a request to a user to reset their password, an email is correctly sent to the user, directing them to reset their password, and providing a link to TD Console.

  • In the query editor, when viewing in a saved query and you select Save As, you are asked for name the new query and redirected to a new query page. Your edits are saved to the new query and don’t affect the original query.

Required Upgrade Presto 317 in Production

Treasure Data’s Presto release is based on the Presto 317 open source release. Our previous release was based on Presto 0.205.

We have done internal testing to identify compatibility and performance regression issues.

For your testing, Presto 0.205 and Presto 317 are available side-by-side to allow testing of the code before the upgrade.

Refer to upgrade details for guidance. Contact our Support team, if you have additional questions.

Recommend Upgrading to TD Toolbelt v0.16.8

If you use the td import: command, we recommend that you upgrade your version of the TD Toolbelt. For instructions on how to validate your version and upgrade, see TD Toolbelt for CLI Use.

Upgrading resolves import failures. 

  • No labels