Treasure Data provides td-agent to collect server-side logs and events, to stream data from .NET applications.


Prerequisites

  • Basic knowledge of .NET

  • Basic knowledge of Treasure Data

Installing td-agent

Install td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.


td-agent receives the records via TCP/HTTP, buffers the records, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

td-agent Install Options

To install td-agent, run one of the following commands based on your environment. The agent program is installed automatically by using the package management software for each platform like rpm/deb/dmg.

RHEL/CentOS 5,6,7

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

Ubuntu and Debian

# 18.04 Bionic
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
# 16.04 Xenial (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
Legacy support for EOL versions is still available
# 14.04 Trusty
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent3.sh | sh
# Debian Stretch (64-bit only) $ curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
# Debian Jessie (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh
# Debian Squeeze (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh

Amazon Linux

You can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on AWS Linux.

MacOS X 10.11+

$ open 'https://td-agent-package-browser.herokuapp.com/3/macosx/td-agent-3.1.1-0.dmg'

MacOS X 10.11.1 (El Capitan) introduced some security changes. After the td-agent is installed, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.

Windows Server 2012+

The Windows installation requires the steps detailed in:

Opscode Chef Repository

You can read more about the repository.

$ echo 'cookbook "td-agent"' >> Berksfile
$ berks install

AWS Elastic Beanstalk is also supported. Windows is not supported.

Modifying /etc/td-agent/td-agent.conf

Specify your API key by setting the apikey option in your /etc/td-agent/td-agent.conf file.

# Input from HTTP
<source>
  type http
  port 8888
</source>

# Treasure Data Output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>

YOUR_API_KEY should be your actual API key string. You can retrieve your API key from your profile in TD Console. Using a [write-only API key](access-control#rest-apis-access) key is recommended.

Restart your agent after the following lines are added:

# Linux
$ sudo /etc/init.d/td-agent restart

# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

td-agent accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically uploads the data into the cloud.

Post the Records via HTTP

Use the following code snippets to post the records to the local Treasure Agent via HTTP.

using System;
using System.Net;
using System.IO;
using System.Text;
using System.Web;

namespace CSharpTreasureAgentExample
{
  class MainClass
  {
    public static void Main (string[] args)
    {
      string json_body = "{\"action\":\"login\",\"user\":2}";
      string tag = "td.production.login";

      // Prepare HTTP Request Object
      HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:8888/" + tag);
      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";
      json_body = "json=" +  System.Web.HttpUtility.UrlEncode(json_body);
      byte[] bytes = Encoding.ASCII.GetBytes(json_body);
      request.ContentLength = bytes.Length;

      // Write the Request
      Stream stream = request.GetRequestStream();
      stream.Write(bytes, 0, bytes.Length);
      stream.Flush();
      stream.Close();

      // Receive the Response
      HttpWebResponse response = (HttpWebResponse)request.GetResponse();
      if (response.StatusCode == HttpStatusCode.OK) {
        using (StreamReader reader = new StreamReader (response.GetResponseStream ())) {
          Console.WriteLine (reader.ReadToEnd ());
        }
      }
      response.Close();
    }
  }
}

Confirming Data Import

Execute your program.

Sending a SIGUSR1 signal will flush td-agent’s buffer. The upload starts immediately.

# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`

# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`

From TD Console

To confirm that your data has been uploaded successfully, check your dataset in the TD Console, Databases view.

From CLI

Or, issue the td tables command if you have a CLI client.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| production | login      | log  | 1         |
+------------+------------+------+-----------+

Production Deployments

High-Availability Configurations of td-agent

For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.

Monitoring td-agent

Monitoring td-agent itself is also important. For general monitoring methods for td-agent, see Monitoring td-agent.

td-agent is fully open-sourced under the Fluentd project.

  • No labels