Fluentd
Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.
Fluentd having two different configurations parts, which will be doing from Elasticsearch and Kibana server side that is known as Fluentd Aggregator Configuration and one more will be application side fluentd configuration which will forward the application logs from app server/webserver to Elasticstack which is known as Fluentd Forwarder Configuration. For Elasticstack configuration with Elasticsearch click here and for kibana click here.
Fluentd Aggregator configuration
Step 1: Install the td-agent
# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
# yum -y install gcc libcurl-devel
# yum groupinstall "Development Tools" kernel-devel kernel-headers -y
# sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
# wget https://rubygems.org/gems/fluent-plugin-elasticsearch/versions/3.3.0
Step 2: Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.
# vim /etc/td-agent/td-agent.conf
<source>
@type forward
port 24224
</source>
<match *.log>
@type copy
<store>
@type file
path /var/log/td-agent/httpd/access_forward.log
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%dT%H%M%S%z
compress gzip
utc
</store>
<store>
@type elasticsearch_dynamic
host 192.168.0.34
port 9200
index_name fluentd-${tag_parts[1]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
logstash_format true
time_format %Y-%m-%dT%H:%M:%S
timezone +0530
include_timestamp true
type_name fluentd
<buffer>
flush_interval 5s
flush_thread_count 3
chunk_limit_size 64m
</buffer>
</store>
</match>
Step 3: Enable and start the td-agent.service
# systemctl enable td-agent.service
# systemctl start td-agent.service
# systemctl status td-agent.service
Step 4: Check the td-agent log file.
# tail -f /var/log/td-agent/td-agent.log
Fluentd Forwarder Configuration:
Step 5: Install the td-agent
# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
# yum -y install gcc libcurl-devel
# yum groupinstall "Development Tools" kernel-devel kernel-headers -y
Step 6: Edit the Log file permissions
i. Change the httpd log directory permissions to “og+rx”
ii. Change the log file permissions to “og+r” in httpd directory
Step 7: Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.
# vim /etc/td-agent/td-agent.conf
<match td.*.*>
@type tdlog
apikey YOUR_API_KEY
auto_create_table
buffer_type file
buffer_path /var/log/td-agent/buffer/td
<secondary>
@type file
path /var/log/td-agent/failed_records
</secondary>
</match>
<match debug.**>
@type stdout
</match>
<source>
@type forward
port 24224
</source>
<source>
@type http
port 8888
</source>
<source>
@type debug_agent
bind 192.168.0.22
port 24230
</source>
<source>
@type tail
path /var/log/httpd/*.log
pos_file /var/log/td-agent/access.log.pos
tag access.log
format none
time_format %Y-%m-%d %H:%M:%S,%L %z
timezone +0530
time_key time
keep_time_key true
types time:time
</source>
<match *.log>
@type copy
<store>
@type file
path /var/log/td-agent/httpd/access_forward.log
</store>
<store>
@type forward
heartbeat_type tcp
<server>
host 192.168.0.34
</server>
flush_interval 5s
</store>
</match>
Step 8: Enable and start the td-agent.service
# systemctl enable td-agent.service
# systemctl start td-agent.service
# systemctl status td-agent.service
Step 11: Check the td-agent log file.
# tail -f /var/log/td-agent/td-agent.log
No comments:
Post a Comment