Data De-duplication for Treasure Agent
Treasure Data supports data de-duplication via the following mechanism:
- Treasure Agent assigns a universally unique identifier (UUID) to each chunk of data.
- Treasure Agent retries whenever it detects network failure. However, this can sometimes result in the same chunk of data being sent more that once.
- When a chunk arrives, to avoid duplication, Treasure Data’s API endpoint inspects the chunk’s ID and discards it if it has been processed in the last 10 minutes.
Last modified: Jun 25 2015 00:31:58 UTC