Treasure Data filter functions are important in ensuring that all of the data coming from various sources can be transformed and unified into a single database ready for querying and segmenting.
Generally, businesses want to transform data to make it compatible with other data, move it to another system, join it with other data, or aggregate information in the data. As organizations bring in data from various sources, data transformation is used to unify the data into a single database.
You are moving your data to a new data store; for example, you are moving to a cloud data warehouse and you need to change the data types.
You want to join unstructured data or streaming data with structured data so you can analyze the data together.
You want to add information to your data to enrich it, such as performing lookups, adding geolocation data, or adding timestamps.
You want to perform aggregations, such as comparing sales data from different regions or totaling sales from different regions.
You want to mask data to protect privacy. Filter functions allow you to mask sensitive data as you bring it from one data source into another.
Filter Functions allow data transformation and unification to occur. The filter determines how your data is filtered as it is moved into or out of Treasure Data based on criteria that you specify. Although filter functions are essentially plugins, they are included automatically with Treasure Data.
They can be used on data import and on data export.
Some of the data connectors and integrations that are provided by Treasure Data already include some filter functionality. Treasure Data also provides several functions that can be added to the YAML file to customize data connectors that you create. The filter functions enable you to customize data manipulation, data masking, and data transfer.
Data manipulation is important in bringing various data sources together and unifying them. Treasure Data filter functions provide the ability to do this with the following:
One increasingly important goal with data transformation is data masking. With privacy becoming increasingly important to organizations, the ability to mask sensitive data as needed is a priority. Treasure Data filter functions provide that level of security using the following:
Filter functions used in data transfer can make the process more effective. It can allow easier data transfer by easily decoding the data and parsing the data. Treasure Data filter functions make data transfer more effective with the following: