Treasure Data provides td-agent to collect server-side logs and events, to stream data from .NET applications.
- Basic knowledge of .NET
- Basic knowledge of Treasure Data
If your security policy requires IP whitelisting, you must add Treasure Data's IP addresses to your allowlist to ensure a successful connection.
Please find the complete list of static IP addresses, organized by region, at the following link:
https://api-docs.treasuredata.com/en/overview/ip-addresses-integrations-result-workers/
If you are unsure of your account's region or require further assistance, please contact your Customer Success representative or our Technical Support team.
Install td-agenton your application servers. td-agent sits within your application servers, focusing on uploading application logs to the cloud.
td-agent receives the records via TCP/HTTP, buffers the records, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.
To install td-agent, run one of the following commands based on your environment. The agent program is installed automatically by using the package management software for each platform like rpm/deb/dmg.
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh# 18.04 Bionic
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
# 16.04 Xenial (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh# 14.04 Trusty
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent3.sh | sh
# Debian Stretch (64-bit only) $ curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
# Debian Jessie (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh
# Debian Squeeze (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | shYou can choose Amazon Linux 1 or Amazon Linux 2. Refer to Installing td-agent on AWS Linux.
$ open 'https://td-agent-package-browser.herokuapp.com/3/macosx/td-agent-3.1.1-0.dmg'MacOS X 10.11.1 (El Capitan) introduced some security changes. After the td-agent is installed, edit the /Library/LaunchDaemons/td-agent.plist file to change /usr/sbin/td-agent to /opt/td-agent/usr/sbin/td-agent.
The Windows installation requires the steps detailed in:
You can read more about the repository.
$ echo 'cookbook "td-agent"' >> Berksfile
$ berks installAWS Elastic Beanstalk is also supported. Windows is not supported.
Specify your API key by setting the apikey option in your /etc/td-agent/td-agent.conf file.
# Input from HTTP
source
type http
port 8888
</source>
# Treasure Data Output
<match td.*.*>
type tdlog
endpoint api.treasuredata.com
apikey YOUR_API_KEY
auto_create_table
buffer_type file
buffer_path /var/log/td-agent/buffer/td
use_ssl true
</match>YOUR_API_KEY should be your actual API key string. You can retrieve your API key from your profile in TD Console. Using a write-only API key is recommended.
Restart your agent after the following lines are added:
# Linux
$ sudo /etc/init.d/td-agent restart
# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plisttd-agent accepts data via port 24224, buffers the data (var/log/td-agent/buffer/td), and automatically uploads the data into the cloud.
Use the following code snippets to post the records to the local Treasure Agent via HTTP.
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Web;
namespace CSharpTreasureAgentExample
{
class MainClass
{
public static void Main (string[] args)
{
string json_body = "{\"action\":\"login\",\"user\":2}";
string tag = "td.production.login";
// Prepare HTTP Request Object
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:8888/" + tag);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
json_body = "json=" + System.Web.HttpUtility.UrlEncode(json_body);
byte[] bytes = Encoding.ASCII.GetBytes(json_body);
request.ContentLength = bytes.Length;
// Write the Request
Stream stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length);
stream.Flush();
stream.Close();
// Receive the Response
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) {
using (StreamReader reader = new StreamReader (response.GetResponseStream ())) {
Console.WriteLine (reader.ReadToEnd ());
}
}
response.Close();
}
}
}Execute your program.
Sending a SIGUSR1 signal will flush td-agent’s buffer. The upload starts immediately.
# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`
# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`To confirm that your data has been uploaded successfully, check your dataset in the TD Console, Databases view.
Or, issue the td tables command if you have a CLI client.
$ td tables
+------------+------------+------+-----------+
| Database | Table | Type | Count |
+------------+------------+------+-----------+
| production | login | log | 1 |
+------------+------------+------+-----------+For high-traffic websites (more than 5 application nodes), use a high availability configuration of td-agent to improve data transfer reliability and query performance.
Monitoring td-agent itself is also important. For general monitoring methods for td-agent, see Monitoring td-agent.
td-agent is fully open-sourced under the Fluentd project.