Treasure Data integrations have a decoder plugin for gzip and bzip2. You can use compressed source data file using this plugin. You can also use a single_file_in_zip decoder for specific integrations.

Configuration

Add the decoders section with the type of compression of your load.yml.

  • gzip

in:
...
  decoders:
  - {type: gzip }
out:
...
  • bzip2

in:
...
  decoders:
  - {type: bzip2 }
out:
...

bzip2 is a format to compress only a byte sequence, not to archive multiple files into a single file. One simple bzip2 file must contain only one file by its definition.


  • single_file_in_zip for only SFTP, FTP, and Amazon S3

The integration for SFTP/FTP and Amazon S3 supports a decoder for single zip files. When you configure your SFTP/FTP or Amazon S3 connector from the CLI (TD Toolbelt), add two lines in the configuration file:

in:
...
  decoders:
  - {type: single_file_in_zip }
out:
...
  • No labels