remove_columns filter plugin for Data Connector

The remove_columns filter plugin for Data Connector expands columns having json into multiple columns. This filter takes five options as explained below:

Option type description required?
remove_columns array An array of names of columns that it removes from schema. [] by default
keep_expanding_json_column array An array of names of columns that it keeps in schema. [] by default
accept_unmatched_columns boolean If true, skip columns that aren’t included in schemas. false by default

Table of Contents

Configuration

Add the filter section with type remove_columns to your load.yml, for example:

in:
...

# This configuration removes "_c0" and "_c1" named columns from schema.
filters:
...
- type: remove_columns
  remove: ["_c0", "_c1"]

out:
...

Last modified: Feb 01 2017 00:09:16 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.