PUSHCODE is a popular Japanese web push service that delivers the most relevant content at the right time to individual site visitors.
This import integration for PUSHCODE enables you to import the following data provided by PUSHCODE from Google Cloud Storage:
- Subscriber List
- Segment List
- Push Result
This topic includes:
Prerequisites
Basic Knowledge of Treasure Data
Basic knowledge of PUSHCODE
- Basic Knowledge of Import Connector for Google Cloud Storage
Requirements and Limitations
- PUSHCODE account
- Enable Treasure Data Integration setting in PUSHCODE
- Files of Subscriber List / Segment List / Push Result are created every day.
Obtain PUSHCODE GCS information
- Navigate to to PUSHCODE Console > Account Settings.
- Enable the Treasure Data integration.
Once you configure the Treasure Data integration, following information is provided.
- Allocated Google Cloud Storage Bucket Name
- Google Cloud Storage service account key (JSON file)
Keep them available for the settings on Treasure Data Console.
Import from PUSHCODE via TD Console
Create Authentication
The PUSHCODE integration uses Google Cloud Storage integration. Follows the steps in the Google Cloud Storage Import Integration to create an authentication using a JSON key file.
Create a Source
1. Open TD Console.
2. Navigate to Integrations Hub > Authentications.
3. Locate your new authentication and select New Source.
Create a Connection
Parameter | Description |
---|---|
Data Transfer Name | You can define the name of your transfer. |
Authentication | The authentication name that is used to a transfer. |
1. Type a source name in the Data Transfer Name field.
2. Select Next.
The Create Source page displays with the Source Table tab selected.
Identify a Source Table
- Select Next.
- The source Table dialog opens, edit the following parameters based on the target PUSHCODE data.
Import Target List | Parameter | Description |
---|---|---|
Subscriber List | Bucket | Bucket Name is found in PUSHCODE console |
Path Prefix | "subscriber_list/subscriber_list_" | |
Path Regex | "subscriber_list\/subscriber_list_[0-9]{12}\.csv\.gz$" | |
Start after path | In order to import only the latest file 'subscriber_list/subscriber_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank. | |
Incremental? | Check this if you want incremental loading. | |
Segment List | Bucket | Bucket Name is found in PUSHCODE console |
Path Prefix | "segment_list/segment_list_" | |
Path Regex | "segment_list/segment_list_[0-9]{12}\.csv\.gz$" | |
Start after path | In order to import only the latest file 'segment_list/segment_list_YYYYMMDD0000.csv.gz', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank. | |
Incremental? | Checked | |
Push Result | Bucket | Bucket Name is found in PUSHCODE console |
Path Prefix | "push_result/push_result_" | |
Path Regex | "push_result\/push_result_[0-9]{12}\.csv\.gz$" | |
Start after path | n order to import only the latest file '"push_result\/push_result_[0-9]{12}\.csv\.gz$', where YYYY is year, MM is month, and DD is the day of the month (zero-padded). To import all past data, leave it blank. | |
Incremental? | Checked |
Define Data Settings
The imported file formats are as follows. You don't need to manually define data settings as the connector automatically guesses these formats.
Import Target List | Name | Description | Type |
---|---|---|---|
Subscriber list | domain_id | your configured site id | int |
domain | your configured site domain | string | |
user_id | User ID who subscribed | int | |
browser_id | Browser ID where subscribed | int | |
user_agent | Browser User Agent | string | |
subscribe_time | When user subscribed | string | |
status | 0: Subscribed / 1: Denied | int | |
unsubscribe_time | When user denied (if user haven't denied, this value is not added) | string | |
Segment list | list_id | User list id | int |
list_name | User list name | string | |
user_id | User ID who is delivered | string | |
Push Result | domain_id | your configured site id | int |
domain | your configured site domain | string | |
push_scenario_inst_i | Push Notification ID | int | |
browser_id | Browser ID where subscribed | int | |
user_id | User ID who subscribed | int | |
push_create_time | When PUSHCODE regsitered a push notification | string | |
push_sent_time | When PUSHCODE sent a push notification | string | |
push_view_time | When user viewed a push notification | int | |
push_click_time | When user clicked a push notification | string |
Select Next.
Preview Your Data
You can see a preview of your data before running the import. The data that displays in the data preview is approximated from your source. It is not the actual data that is imported. Data preview is optional and you can safely skip to the next page of the dialog if you want.
- To preview your data, select Generate Preview. Optionally, select Next.
- Verify that the data meets your expectations.
- Select Next.
Define Your Data Placement
Select the target database and table where you want your data placed, and then indicate how often the import should run.
- Select Data Placement.
- Select a Database > Select an existing or Create New Database. Optionally, enter a database name.
- Select a Table> Select an existing or Create New Table. Optionally, type a table name.
- Choose the method for importing the data.
- Append (default): Data import results are appended to the table.
If the table does not exist, it will be created. - Always Replace: Replaces the entire content of an existing table with the result output of the query. If the table does not exist, a new table is created.
- Replace on New Data: Only replace the entire content of an existing table with the result output when there is new data.
- Append (default): Data import results are appended to the table.
- Select the Timestamp-based Partition Key column.
If you want to set a different partition key seed than the default key, you can specify the long or timestamp column as the partitioning time. As a default time column, it uses upload_time with the add_time filter. - Select the Timezone for your data storage.
- Choose when and how often you want to run this query:
- Run once:
- Select Off.
- Select Scheduling Timezone.
- Select Create & Run Now.
- Repeat the query:
- Select On.
- Select the Schedule. The UI provides these four options: @hourly, @daily, and @monthly or custom cron.
- You can also select Delay Transfer and add a delay of execution time.
- Select Scheduling Timezone.
- Select Create & Run Now.
- Run once:
To see the results of your transfer, go to Data Workbench > Databases.
Import from PUSHCODE via Workflow
You can import data from PUSHCODE by using td_load>: operator of workflow. If you have already created a SOURCE, you can run it; if you don't want to create a SOURCE, you can import it using a YAML file.
Using a Source or YAML File
Using a Source
1. Identify your source.
2. To obtain a unique ID, open the Source list and search for the source you created.
3. Open the menu and select "Copy Unique ID".
4. Define a workflow task using td_load> operator.
+load: td_load>: unique_id_of_your_source database: ${td.dest_db} Table: ${td.dest_table}
5. Run a workflow.
Using a yml file
1. Identify your yml file. If you need to create the yml file, review Google Cloud Storage Import Integration for reference.
2. Define a workflow task using td_load> operator.
+load: td_load>: config/daily_load.yml database: ${td.dest_db} Table: ${td.dest_table}
3. Run a workflow
Sample Workflow Code
Visit Treasure Boxes for sample workflow code.
Import from PUSHCODE via CLI (Toolbelt)
Before setting up the connector, install the most current TD Toolbelt.
PUSHCODE integration with CLI uses Google Cloud Storage integration. Follows steps in the Google Cloud Storage Import Integration to create an authentication with selecting JSON key file.
See Define Data Settings to obtain required parameters.