Treasure Data lets you work with a wide variety of industry data sources, including relational databases, file data sources, application data sources, and so on. You can import data into Treasure Data by uploading a CSV or TSV file or from an external tool using a data connector. You can transfer data in bulk or can configure your account to import data incrementally. After establishing your data transfer, you can manage data input as a job.
The primary types of data ingestion are:
- Batch Ingest — batch ingest is initiated by you or your data engineer or your database admin. It can be historical (bulk import) or incremental (scheduled). Batch ingest is initiated independent of any action your customers may take.
- Real-Time Ingest — real time ingest happens when a specific event happens on your website or in a tool or external database that Treasure Data is integrated with. Beyond specifying the trigger event that will initiate the real-time ingestion, you have no control over when the ingest happens.
- Live Connect Ingest — Live Connect ingest imports data from data wharehouses using either Easy Sync or zero-copy.
You can stream or bulk load data into the platform by setting up an integration. Treasure Data offers data connections for website tracking, mobile and app, social media, and much more.
When you configure an integration, you provide authentication to access the data source. You can see a list of all your configured and saved integrations under Integrations Hub > Authentications.
After you create and configure the authentication, you can create data sources from this page that specify the specific source, the schedule for import, and also specify where in Treasure Data to put the incoming data.
You can see your data sources under Integrations Hub > Sources. You can edit your sources
