# Apache Access Logs Import Integration

td-agent was discontinued in December 2023 and has been replaced by **fluent-package**. The fluent-package is the official successor maintained by the [Cloud Native Computing Foundation](https://www.cncf.io/projects/).

You can tail Apache logs using Fluentd to continuously import the access logs into the cloud.

## Installing Fluentd

## Fluentd (fluent-package) Install Options

td-agent was discontinued in December 2023 and has been replaced by **fluent-package**. The fluent-package is the official successor maintained by the [Cloud Native Computing Foundation](https://www.cncf.io/projects/). For migration guidance from td-agent, see [Fluentd Installation Guide](https://docs.fluentd.org/installation/install-fluent-package).

To install `fluent-package`, run one of the following commands based on your environment.

### RHEL/CentOS/Rocky Linux


```bash
# fluent-package 6 LTS (recommended)
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-redhat-fluent-package6-lts.sh | sh
```

### Ubuntu


```bash
# Ubuntu 24.04 Noble - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package6-lts.sh | sh

# Ubuntu 22.04 Jammy - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package6-lts.sh | sh
```

### Debian


```bash
# Debian Bookworm - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-debian-bookworm-fluent-package6-lts.sh | sh
```

### Amazon Linux


```bash
# Amazon Linux 2023 - fluent-package 6 LTS
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-amazon2023-fluent-package6-lts.sh | sh
```

### Windows

Download the MSI installer from:

- [fluent-package 6 LTS for Windows](https://fluentd.cdn.cncf.io/lts/6/windows/index.html)


After installation:

1. Edit the configuration file at `C:/opt/fluent/etc/fluent/fluentd.conf`
2. Start the service using `net start fluentdwinsvc` or via Services administrative tool


### macOS

fluent-package for macOS is planned to be available via Homebrew. For current installation options, see [Fluentd Installation Guide](https://docs.fluentd.org/installation).

## Starting the Service

After installation, start and verify the Fluentd service.

### Linux


```bash
sudo systemctl start fluentd.service
sudo systemctl status fluentd.service
```

The configuration file is located at `/etc/fluent/fluentd.conf`.

### Windows


```cmd
net start fluentdwinsvc
```

The configuration file is located at `C:\opt\fluent\etc\fluent\fluentd.conf`.

### macOS (gem installation)


```bash
fluentd -c /path/to/fluentd.conf
```

For more details, see the [Fluentd Documentation](https://docs.fluentd.org/).

## Modifying fluentd.conf

Specify your authentication key by setting the `apikey` option. You can view your API key from the TD Console.

To modify the file, go to `/etc/fluent/fluentd.conf` (for fluent-package). Set the `apikey` option in your fluentd.conf file.

Where *YOUR_API_KEY* is your actual apikey string.


```conf
# Tailing the Apache Log
<source>
  @type tail
  path /var/log/httpd-access.log
  pos_file /var/log/fluent/httpd-access.pos
  tag td.production.access
  <parse>
    @type apache2
  </parse>
</source>

# Treasure Data Input and Output
<match td.*.*>
  @type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  use_ssl true
  <buffer>
    @type file
    path /var/log/fluent/buffer/td
  </buffer>
</match>
```

Restart the Fluentd service when the following lines are in place.


```bash
sudo systemctl restart fluentd.service
```

Fluentd keeps tailing the log, buffers the log (`/var/log/fluent/buffer/td`), and automatically uploads the log into the cloud.

## Confirming Data Import

Sending a SIGUSR1 signal flushes Fluentd's buffer and the upload starts immediately.


```bash
# generate access logs
$ curl http://host:port/

# flush the buffer
$ kill -USR1 $(cat /var/run/fluent/fluentd.pid)
```

To confirm that your data has been uploaded successfully, issue the `td tables` command as follows.


```
$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| production | access     | log  | 1         |
+------------+------------+------+-----------+
```

Check `/var/log/fluent/fluentd.log` if it's not working correctly. The `fluentd:fluentd` user must have permission to read the logs.

**Fluentd handles log-rotation**. Fluentd keeps a record of the last position of the log, ensuring that each line is read exactly one time even if the Fluentd process goes down. However, because the information is kept in a file, the "exactly once" guarantee breaks down if the file becomes corrupted.

## Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.

- [Schema Management](/products/customer-data-platform/data-workbench/databases/schema-management)
- [Writing Hive Queries](/products/customer-data-platform/data-workbench/queries/hive/writing_hive_queries)