Skip to content
Tools
/
/
Php Apps Import
Last updated

PHP Apps Import Integration

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.

Treasure Data provides Fluentd to collect server-side logs and events, and to seamlessly import the data from PHP applications.

Prerequisites

  • Basic knowledge of PHP.
  • Basic knowledge of Treasure Data.
  • PHP 8.1 or higher (for local testing).

Installing Fluentd

Install Fluentd (fluent-package) on your application servers. Fluentd sits within your application servers, focusing on uploading application logs to the cloud.

The fluent-logger-php library enables PHP applications to post records to their local Fluentd. Fluentd, in turn, uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

Fluentd (fluent-package) Install Options

td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation. For migration guidance from td-agent, see Fluentd Installation Guide.

To install fluent-package, run one of the following commands based on your environment.

RHEL/CentOS/Rocky Linux

# fluent-package 6 LTS (recommended)
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-redhat-fluent-package6-lts.sh | sh

Ubuntu

# Ubuntu 24.04 Noble - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package6-lts.sh | sh

# Ubuntu 22.04 Jammy - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package6-lts.sh | sh

Debian

# Debian Bookworm - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-debian-bookworm-fluent-package6-lts.sh | sh

Amazon Linux

# Amazon Linux 2023 - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-amazon2023-fluent-package6-lts.sh | sh

Windows

Download the MSI installer from:

After installation:

  1. Edit the configuration file at C:/opt/fluent/etc/fluent/fluentd.conf
  2. Start the service using net start fluentdwinsvc or via Services administrative tool

macOS

fluent-package for macOS is planned to be available via Homebrew. For current installation options, see Fluentd Installation Guide.

Starting the Service

After installation, start and verify the Fluentd service.

Linux

sudo systemctl start fluentd.service
sudo systemctl status fluentd.service

The configuration file is located at /etc/fluent/fluentd.conf.

Windows

net start fluentdwinsvc

The configuration file is located at C:\opt\fluent\etc\fluent\fluentd.conf.

macOS (gem installation)

fluentd -c /path/to/fluentd.conf

For more details, see the Fluentd Documentation.

Modifying fluentd.conf

Next, specify your API key by setting the apikey option in your /etc/fluent/fluentd.conf file (for fluent-package). You can view your API key from your profile in the TD Console.

# Unix Domain Socket Input
<source>
  @type unix
  path /var/run/fluent/fluentd.sock
</source>

# Treasure Data Output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>

YOUR_API_KEY should be your actual apikey string. You can retrieve your API key from your profiles in TD Console. Using a write-only API key is recommended.

Restart the Fluentd service when the following lines are in place.

# Linux
sudo systemctl restart fluentd.service

# macOS
# Restart Fluentd using the method you used to start it.
# Examples:
# - If you run Fluentd manually (Ruby gem), stop the running process and start it again, e.g.:
#     fluentd -c /etc/fluent/fluentd.conf
# - If you configured a launchd service or another service manager, restart that service.

Fluentd now accepts data via the Unix socket, buffers the data (/var/log/fluent/buffer/td), and automatically uploads the data into the cloud.

Using fluent-logger-php

To use fluent-logger-php, use Composer as a package manager. First, create composer.json in your directory with the following content.

{
  "require": {
    "fluent/logger": "v1.0.0"
  }
}

Then, install Composer and install necessary libraries.

$ curl -sS https://getcomposer.org/installer | php
$ php composer.phar install

Next, initialize and post the records as follows.

<?php
require_once __DIR__.'/vendor/autoload.php';
use Fluent\Logger\FluentLogger;
$logger = new FluentLogger("unix:///var/run/fluent/fluentd.sock");
$logger->post("td.test_db.test_table", array("hello"=>"world"));
$logger->post("td.test_db.follow", array("from"=>"userA", "to"=>"userB"));

Confirming Data Import

Execute the preceding program.

$ php test.php

Sending a SIGUSR1 signal flushes Fluentd's buffer. The upload starts immediately.

# Linux
$ kill -USR1 $(cat /var/run/fluent/fluentd.pid)

# macOS (gem installation)
# Send SIGUSR1 to the Fluentd process

To confirm that your data has been uploaded successfully, issue the td tables command as follows:

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| test_db    | test_table | log  | 1         |
| test_db    | follow     | log  | 1         |
+------------+------------+------+-----------+

The first argument of post() determines the database name and table name. If you specify td.test\_db.test\_table, the data is imported into the table *test_table* within the database *test_db*. They are automatically created at upload time.

Tips on Production Deployment

Use Apache and mod_php

We recommend that you use Apache and mod_php. Other setups have not been fully validated.

Use Apache prefork MPM

Use Apache prefork MPM. Other MPMs such as worker MPM should not be used. You can confirm your current settings with the apachectl -V command.

$ apachectl -V | grep MPM:
Server MPM:     Prefork

Set MaxRequestsPerChild

We recommend that you periodically restart your PHP processes by setting MaxRequestsPerChild in your Apache conf.

<IfModule mpm_prefork_module>
  StartServers          32
  MinSpareServers       32
  MaxSpareServers       32
  MaxClients            32
  MaxRequestsPerChild 4096
</IfModule>

Do not set MaxRequestsPerChild to zero.

Production Deployments

High-Availability Configurations of Fluentd

For high-traffic websites (more than 5 application nodes), use a high availability configuration of Fluentd to improve data transfer reliability and query performance.

Monitoring Fluentd

Monitoring Fluentd itself is also important. Refer to this document for general monitoring methods for Fluentd:

Fluentd is fully open-sourced under the Fluentd project.

FAQs

“Resource temporarily unavailable” warning message appears in my PHP application

This problem occurs when you have either a relatively high volume, or an old Linux kernel version. You must tune up the Linux kernel a little bit.

Increase Max # of File Descriptors

First, increase the max number of file descriptors per process. When you type ulimit -n command and the result shows 1024, complete the follow instructions to increase to 65535:

Optimize Kernel Parameters

Add the following parameters to your /etc/sysctl.conf file. Either type sysctl -w or reboot your node to have the changes take effect. You need a root permission.

net.core.somaxconn = 1024
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240    65535

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive and Trino Query Languages.

Previous page
Next page