Data De-duplication for Treasure Agent

Treasure Data supports data de-duplication via the following mechanism:

  1. Treasure Agent assigns a universally unique identifier (UUID) to each chunk of data.
  2. Treasure Agent retries whenever it detects network failure. However, this can sometimes result in the same chunk of data being sent more that once.
  3. When a chunk arrives, to avoid duplication, Treasure Data’s API endpoint inspects the chunk’s ID and discards it if it has been processed in the last 10 minutes.

Last modified: Jun 25 2015 00:31:58 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.