td-agent was discontinued in December 2023 and has been replaced by fluent-package. The fluent-package is the official successor maintained by the Cloud Native Computing Foundation.
Fluentd sometimes gets a "Failed to upload to TreasureData: Import failed" error. There are several possible causes:
- Your network becomes unstable during the upload.
- Treasure Data's API shut down the connection by an unexpected network/node condition.
Fluentd has a robust retry mechanism for these cases, so continue to import the data to Treasure Data. You should see the message "retry succeeded" in your logs after several retries.
If you continue to get the same error and it exceeds the retry limit, it is a non-network problem. Contact support@treasure-data.com.
Here are some common reasons that you cannot import data.
- Make sure your network is live and allows you to access external networks.
- If your data contains numbers greater than 2e64, you need to upgrade your fluent-plugin-td to the most recent version.
There is some agent overhead associated with running Fluentd. The resource consumption is approximate as follows:
- Resident memory (actual RAM used): 50MB
- CPU Runtime: less than 2% of averaged runtime + your workload
- Disk: Linux 120MB + your file buffer (configurable)
If you think Fluentd is slow, see "5 Tips to Optimize Fluentd Performance".
If you are having issues, you can enable verbose logging by editing the systemd service or passing the -vv flag to Fluentd.
For fluent-package, edit the configuration or start Fluentd manually with verbose logging:
fluentd -c /etc/fluent/fluentd.conf -vvYou can now find more verbose logs in /var/log/fluent/fluentd.log.
You might see the following message in the Fluentd log.
tdlog plugin will change the API endpoint from api.us01.treasure-data.com to api.us01.treasuredata.com
If you want to keep api.treasure-data.com, set 'endpoint api.treasure-data.com' in tdlog configurationThis message is for users who change the control of network access to api.treasure-data.com. If you don't set access restrictions based on our API endpoint, ignore the warning.