PUSHCODE is a popular Japanese web push service that delivers the most relevant content at the right time to individual site visitors.

This import integration for PUSHCODE enables you to import the following data provided by PUSHCODE from Google Cloud Storage:

  • Subscriber List
  • Segment List
  • Push Result

This topic includes:

Prerequisites

  • Basic Knowledge of Treasure Data

  • Basic knowledge of PUSHCODE

  • Basic Knowledge of Import Connector for Google Cloud Storage

Requirements and Limitations

  • PUSHCODE account
  • Enable Treasure Data Integration setting in PUSHCODE
  • Files of Subscriber List / Segment List / Push Result are created every day.

Obtain PUSHCODE GCS information

  1. Navigate to to PUSHCODE Console > Account Settings.
  2. Enable the Treasure Data integration.

Once you configure the Treasure Data integration, following information is provided.

  • Allocated Google Cloud Storage Bucket Name
  • Google Cloud Storage service account key (JSON file)

Keep them available for the settings on Treasure Data Console.

Import from PUSHCODE via TD Console

Create Authentication

The PUSHCODE integration uses Google Cloud Storage integration. Follows the steps in the Google Cloud Storage Import Integration to create an authentication using a JSON key file.

Create a Source

1. Open TD Console.
2. Navigate to Integrations Hub > Authentications.
3. Locate your new authentication and select New Source.

Create a Connection

ParameterDescription
Data Transfer NameYou can define the name of your transfer.
AuthenticationThe authentication name that is used to a transfer.
1. Type a source name in the Data Transfer Name field.
2. Select Next.

The Create Source page displays with the Source Table tab selected.

Identify a Source Table

  1. Select Next.
  2. The source Table dialog opens, edit the following parameters based on the target PUSHCODE data.
Import Target ListParameterDescription
Subscriber ListBucketBucket Name is found in PUSHCODE console
Path Prefix"subscriber_list/subscriber_list_"
Path Regex"subscriber_list\/subscriber_list_[0-9]{12}\.csv\.gz$"
Start after path

In order to import only the latest file 'subscriber_list/subscriber_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded).

To import all past data, leave it blank.

Incremental?

Check this if you want incremental loading.
Segment ListBucketBucket Name is found in PUSHCODE console
Path Prefix"segment_list/segment_list_"
Path Regex"segment_list/segment_list_[0-9]{12}\.csv\.gz$"
Start after path

In order to import only the latest file 'segment_list/segment_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded).

To import all past data, leave it blank.

Incremental?

Checked
Push ResultBucketBucket Name is found in PUSHCODE console
Path Prefix"push_result/push_result_"
Path Regex"push_result\/push_result_[0-9]{12}\.csv\.gz$"
Start after path

n order to import only the latest file '"push_result\/push_result_[0-9]{12}\.csv\.gz$', where YYYY is year, MM is month, and DD is the day of the month (zero-padded).

To import all past data, leave it blank.

Incremental?

Checked

Define Data Settings

The imported file formats are as follows. You don't need to manually define data settings as the connector automatically guesses these formats.

Import Target ListNameDescriptionType
Subscriber listdomain_idyour configured site idint
domainyour configured site domainstring

user_id

User ID who subscribedint
browser_idBrowser ID where subscribedint
user_agentBrowser User Agentstring
subscribe_timeWhen user subscribedstring
status0: Subscribed / 1: Deniedint
unsubscribe_timeWhen user denied (if user haven't denied, this value is not added)string
Segment listlist_idUser list idint
list_nameUser list namestring

user_id

User ID who is deliveredstring
Push Resultdomain_idyour configured site idint
domainyour configured site domainstring

push_scenario_inst_i

Push Notification IDint
browser_idBrowser ID where subscribedint

user_id

User ID who subscribedint
push_create_timeWhen PUSHCODE regsitered a push notificationstring
push_sent_timeWhen PUSHCODE sent a push notificationstring
push_view_timeWhen user viewed a push notificationint
push_click_timeWhen user clicked a push notificationstring

Select Next.

Preview Your Data

You can see a preview of your data before running the import. The data that displays in the data preview is approximated from your source. It is not the actual data that is imported. Data preview is optional and you can safely skip to the next page of the dialog if you want.

  1. To preview your data, select Generate Preview. Optionally, select Next.
  2. Verify that the data meets your expectations.
  3. Select Next.

Define Your Data Placement

Select the target database and table where you want your data placed, and then indicate how often the import should run.

  1. Select Data Placement.
  2. Select a Database > Select an existing or Create New Database. Optionally, enter a database name.
  3. Select a Table> Select an existing or Create New Table. Optionally, type a table name.
  4. Choose the method for importing the data.
    • Append (default): Data import results are appended to the table.
      If the table does not exist, it will be created.
    • Always Replace: Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created. 
    • Replace on New Data: Only replace the entire content of an existing table with the result output when there is new data.
  5. Select the Timestamp-based Partition Key column.
    If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter.
  6. Select the Timezone for your data storage.
  7. Choose when and how often you want to run this query:
    • Run once:
      • Select Off.
      • Select Scheduling Timezone.
      • Select Create & Run Now.
    • Repeat the query:
      • Select On.
      • Select the Schedule. The UI provides these four options: @hourly, @daily, and @monthly or custom cron.
      • You can also select Delay Transfer and add a delay of execution time.
      • Select Scheduling Timezone.
      • Select Create & Run Now.

To see the results of your transfer, go to Data Workbench > Databases.

Import from PUSHCODE via Workflow

You can import data from PUSHCODE by using td_load>: operator of workflow. If you have already created a SOURCE, you can run it; if you don't want to create a SOURCE, you can import it using a YAML file.

Using a Source or YAML File 

Using a Source

1. Identify your source.
2. To obtain a unique ID, open the Source list and search for the source you created.
3. Open the menu and select "Copy Unique ID".


4. Define a workflow task using td_load> operator.
+load:
  td_load>: unique_id_of_your_source
  database: ${td.dest_db}
  Table: ${td.dest_table}
5. Run a workflow.

Using a yml file

1. Identify your yml file. If you need to create the yml file, review Google Cloud Storage Import Integration for reference.
2. Define a workflow task using td_load> operator.
+load:
  td_load>: config/daily_load.yml
  database: ${td.dest_db}
  Table: ${td.dest_table}
3. Run a workflow

Sample Workflow Code

Visit Treasure Boxes for sample workflow code.

Import from PUSHCODE via CLI (Toolbelt)

Before setting up the connector, install the most current TD Toolbelt.

PUSHCODE integration with CLI uses Google Cloud Storage integration. Follows steps in the Google Cloud Storage Import Integration to create an authentication with selecting JSON key file.

See Define Data Settings to obtain required parameters.



  • No labels