To avoid duplicate records during a mobile or Unity SDK data transfer, Treasure Data offers per record deduplication.
Our per record deduplication is based on either the UUID field provided by the customer within the record (or an internally assigned UUID provided as metadata).
The SDK imports events in one style with the combination of these features:
- This SDK keeps buffered events by adding unique keys and retries to upload them until confirming the events are uploaded and stored on the server-side (at least once)
- The server side remembers the unique keys of all events within the past 1 hour by default and can prevent duplicate imports.
Deduplication is a best effort system that identifies a duplicate record if a record with the same identifier is seen in the same dataset, within the last hour at most or within the last 4096 records, whichever comes first.