Server-Side Agent with .NET Apps

Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. This article explains 4 steps to streamingly import the data from .NET applications, through Treasure Agent.

Table of Contents

Prerequisites

  • Basic knowledge of .NET.
  • Basic knowledge of Treasure Data.

What is Treasure Agent?

First of all, Treasure Agent (td-agent) needs to be installed on your application servers. Treasure Agent is an agent program sits within your application servers, focusing on uploading application logs to the cloud.



Treasure Agent receives the records via TCP/HTTP, buffers them, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

How to install Treasure Agent?

This video demonstrates how to install Treasure Agent in 3 minutes.

Step 1: Installing Treasure Agent

To install Treasure Agent (td-agent), please execute one of the command below based on your environment. The agent program will be installed automatically by using the package management software for each platform like rpm/deb/dmg.

RHEL/CentOS 5,6,7

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

Ubuntu & Debian

# 14.04 Trusty (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh
# 10.04 Lucid
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-lucid-td-agent2.sh | sh

# Debian Squeeze (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh
# Debian Wheezy (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-wheezy-td-agent2.sh | sh

Amazon Linux

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

MacOS X 10.11+

$ open 'https://packages.treasuredata.com/2/macosx/td-agent-2.3.0-0.dmg'
Untitled-3
With MacOS X 10.11.1 (El Capitan), some security changes were introduced and we are testing the changes we made to td-agent for this version of OS. For now, once the td-agent is installed, please edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.

Windows Server 2012+

Windows installation needs multiple steps to follow. Please go to this documentation.



Opscode Chef (repository)

$ echo 'cookbook "td-agent"' >> Berksfile
$ berks install

AWS Elastic Beanstalk is also supported. Windows is currently NOT supported.

Step 2: Modifying /etc/td-agent/td-agent.conf

Next, please specify your API key by setting the apikey option in your /etc/td-agent/td-agent.conf file.

# Input from HTTP
<source>
  type http
  port 8888
</source>

# Treasure Data Output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>
Untitled-3
YOUR_API_KEY should be your actual apikey string. You can retrieve your api key from HERE. Using a [write-only API key](access-control#rest-apis-access) key is recommended.

Please restart your agent once these lines are in place.

# Linux
$ sudo /etc/init.d/td-agent restart

# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

td-agent will now accept data via port 24224, buffer it (var/log/td-agent/buffer/td), and automatically upload it into the cloud.

Step 3: Post the records via HTTP

Please use the following code snippets to post the records to local Treasure Agent via HTTP.

using System;
using System.Net;
using System.IO;
using System.Text;
using System.Web;

namespace CSharpTreasureAgentExample
{
  class MainClass
  {
    public static void Main (string[] args)
    {
      string json_body = "{\"action\":\"login\",\"user\":2}";
      string tag = "td.production.login";

      // Prepare HTTP Request Object
      HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:8888/" + tag);
      request.Method = "POST";
      request.ContentType = "application/x-www-form-urlencoded";
      json_body = "json=" +  System.Web.HttpUtility.UrlEncode(json_body);
      byte[] bytes = Encoding.ASCII.GetBytes(json_body);
      request.ContentLength = bytes.Length;

      // Write the Request
      Stream stream = request.GetRequestStream();
      stream.Write(bytes, 0, bytes.Length);
      stream.Flush();
      stream.Close();

      // Receive the Response
      HttpWebResponse response = (HttpWebResponse)request.GetResponse();
      if (response.StatusCode == HttpStatusCode.OK) {
        using (StreamReader reader = new StreamReader (response.GetResponseStream ())) {
          Console.WriteLine (reader.ReadToEnd ());
        }
      }
      response.Close();
    }
  }
}

Step 4: Confirming Data Import

Please execute your probram first, then send a SIGUSR1 signal will flush td-agent’s buffer; upload will start immediately.

# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`

# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`

From Web Console

To confirm that your data has been uploaded successfully, check your dataset from the web browser HERE.

From CLI

Or, please issue the td tables command if you have a CLI client.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| production | login      | log  | 1         |
+------------+------------+------+-----------+

Production Deployments

High-Availablability Configurations of td-agent

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

Monitoring td-agent

Monitoring td-agent itself is also important. Please refer to this document for general monitoring methods for td-agent.

Untitled-3
td-agent is fully open-sourced under the fluentd project.

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.


Last modified: Jan 17 2017 23:13:49 UTC

If this article is incorrect or outdated, or omits critical information, please let us know. For all other issues, please see our support channels.