PUSHCODE is a popular Japanese web push service that delivers the most relevant content at the right time to individual site visitors.

This import integration for PUSHCODE enables you to import the following data provided by PUSHCODE from Google Cloud Storage:

Subscriber List
Segment List
Push Result

This topic includes:

Prerequisites

Basic Knowledge of Treasure Data
Basic knowledge of PUSHCODE
Basic Knowledge of Import Connector for Google Cloud Storage

Requirements and Limitations

PUSHCODE account
Enable Treasure Data Integration setting in PUSHCODE
Files of Subscriber List / Segment List / Push Result are created every day.

Obtain PUSHCODE GCS information

Navigate to to PUSHCODE Console > Account Settings.
Enable the Treasure Data integration.

Once you configure the Treasure Data integration, following information is provided.

Allocated Google Cloud Storage Bucket Name
Google Cloud Storage service account key (JSON file)

Keep them available for the settings on Treasure Data Console.

Import from PUSHCODE via TD Console

Create Authentication

The PUSHCODE integration uses Google Cloud Storage integration. Follows the steps in the Google Cloud Storage Import Integration to create an authentication using a JSON key file.

Create a Source

1. Open TD Console.

2. Navigate to Integrations Hub > Authentications.

3. Locate your new authentication and select New Source.

Create a Connection

Parameter	Description
Data Transfer Name	You can define the name of your transfer.
Authentication	The authentication name that is used to a transfer.

1. Type a source name in the Data Transfer Name field.

2. Select Next.

The Create Source page displays with the Source Table tab selected.

Identify a Source Table

Select Next.
The source Table dialog opens, edit the following parameters based on the target PUSHCODE data.

Import Target List	Parameter	Description
Subscriber List	Bucket	Bucket Name is found in PUSHCODE console
	Path Prefix	"subscriber_list/subscriber_list_"
	Path Regex	"subscriber_list\/subscriber_list_[0-9]{12}\.csv\.gz$"
	Start after path	In order to import only the latest file 'subscriber_list/subscriber_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank.
	Incremental?	Check this if you want incremental loading.
Segment List	Bucket	Bucket Name is found in PUSHCODE console
	Path Prefix	"segment_list/segment_list_"
	Path Regex	"segment_list/segment_list_[0-9]{12}\.csv\.gz$"
	Start after path	In order to import only the latest file 'segment_list/segment_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank.
	Incremental?	Checked
Push Result	Bucket	Bucket Name is found in PUSHCODE console
	Path Prefix	"push_result/push_result_"
	Path Regex	"push_result\/push_result_[0-9]{12}\.csv\.gz$"
	Start after path	n order to import only the latest file '"push_result\/push_result_[0-9]{12}\.csv\.gz$', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank.
	Incremental?	Checked

Define Data Settings

The imported file formats are as follows. You don't need to manually define data settings as the connector automatically guesses these formats.

Import Target List	Name	Description	Type
Subscriber list	domain_id	your configured site id	int
	domain	your configured site domain	string
	user_id	User ID who subscribed	int
	browser_id	Browser ID where subscribed	int
	user_agent	Browser User Agent	string
	subscribe_time	When user subscribed	string
	status	0: Subscribed / 1: Denied	int
	unsubscribe_time	When user denied (if user haven't denied, this value is not added)	string
Segment list	list_id	User list id	int
	list_name	User list name	string
	user_id	User ID who is delivered	string
Push Result	domain_id	your configured site id	int
	domain	your configured site domain	string
	push_scenario_inst_i	Push Notification ID	int
	browser_id	Browser ID where subscribed	int
	user_id	User ID who subscribed	int
	push_create_time	When PUSHCODE regsitered a push notification	string
	push_sent_time	When PUSHCODE sent a push notification	string
	push_view_time	When user viewed a push notification	int
	push_click_time	When user clicked a push notification	string

Select Next.

Preview Your Data

You can see a preview of your data before running the import. The data that displays in the data preview is approximated from your source. It is not the actual data that is imported. Data preview is optional and you can safely skip to the next page of the dialog if you want.

To preview your data, select Generate Preview. Optionally, select Next.
Verify that the data meets your expectations.
Select Next.

Define Your Data Placement

Select the target database and table where you want your data placed, and then indicate how often the import should run.

Select Data Placement.
Select a Database > Select an existing or Create New Database. Optionally, enter a database name.
Select a Table> Select an existing or Create New Table. Optionally, type a table name.
Choose the method for importing the data.
- Append (default): Data import results are appended to the table.
  If the table does not exist, it will be created.
- Always Replace: Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created.
- Replace on New Data: Only replace the entire content of an existing table with the result output when there is new data.
Select the Timestamp-based Partition Key column.
If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter.
Select the Timezone for your data storage.
Choose when and how often you want to run this query:
- Run once:
  - Select Off.
  - Select Scheduling Timezone.
  - Select Create & Run Now.
- Repeat the query:
  - Select On.
  - Select the Schedule. The UI provides these four options: @hourly, @daily, and @monthly or custom cron.
  - You can also select Delay Transfer and add a delay of execution time.
  - Select Scheduling Timezone.
  - Select Create & Run Now.

To see the results of your transfer, go to Data Workbench > Databases.

Import from PUSHCODE via Workflow

You can import data from PUSHCODE by using td_load>: operator of workflow. If you have already created a SOURCE, you can run it; if you don't want to create a SOURCE, you can import it using a YAML file.

Using a Source or YAML File

Using a Source

1. Identify your source.

2. To obtain a unique ID, open the Source list and search for the source you created.

3. Open the menu and select "Copy Unique ID".

4. Define a workflow task using td_load> operator.

+load:
  td_load>: unique_id_of_your_source
  database: ${td.dest_db}
  Table: ${td.dest_table}

5. Run a workflow.

Using a yml file

1. Identify your yml file. If you need to create the yml file, review Google Cloud Storage Import Integration for reference.

2. Define a workflow task using td_load> operator.

+load:
  td_load>: config/daily_load.yml
  database: ${td.dest_db}
  Table: ${td.dest_table}

3. Run a workflow

Sample Workflow Code

Visit Treasure Boxes for sample workflow code.

Import from PUSHCODE via CLI (Toolbelt)

Before setting up the connector, install the most current TD Toolbelt.

PUSHCODE integration with CLI uses Google Cloud Storage integration. Follows steps in the Google Cloud Storage Import Integration to create an authentication with selecting JSON key file.

See Define Data Settings to obtain required parameters.

Page tree

PUSHCODE Import Integration