Elasticsearch:
Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine tuned relevancy, and powerful analytics that scale with ease.
Kibana:
Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.
Fluentd:
Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language. Fluentd having two different configurations parts, which will be doing from Elasticsearch and Kibana server side that is known as Fluentd Aggregator Configuration and one more will be application side fluentd configuration which will forward the application logs from app server/webserver to Elasticstack which is known as Fluentd Forwarder Configuration.
Pre-requisites:
System requirements, two virtual machines
1. elasticsearch, kibana, fluentd aggregator (one system) :192.168.0.34
2. In Application Node need to install fluentd forwarder : 192.168.0.22
ElasticSearch Installation:
Step 1: Before installing Elasticsearch, add the
elastic.co key to the server.
$ sudo rpm --import
https://artifacts.elastic.co/GPG-KEY-elasticsearch
Step 2: Now I am downloading
the latest rpm of Elastic Search
$ sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-x86_64.rpm
Step 3: Install the Downloaded RPM.
$ sudo rpm
-ivh elasticsearch-7.2.0-x86_64.rpm
Step 4: Now go to the configuration directory and
edit the elasticsearch.yml configuration file. Enable the below lines in
configuration file
$ sudo cd /etc/elasticsearch/
$ sudo vim elasticsearch.yml
# bootstrap.memory_lock: true
# network.host: 192.168.0.34
# http.port: 9200
Step 5: Now edit the elasticsearch.service file for
the memory lock configuration. Uncomment LimitMEMLOCK line or if it is not
there please add this line to limit session
$ sudo vim
/usr/lib/systemd/system/elasticsearch.service
# LimitMEMLOCK=infinity
Step 6: Edit the sysconfig configuration file for
Elasticsearch. Uncomment line 60 and
make sure the value is 'unlimited'.
$ sudo vim /etc/sysconfig/elasticsearch
# MAX_LOCKED_MEMORY=unlimited
The
Elasticsearch configuration is finished.
Step 7: Reload systemd, enable Elasticsearch to start
at boot time, then start the service.
$ sudo systemctl daemon-reload
$ sudo systemctl enable
elasticsearch
$ sudo systemctl start
elasticsearch
Step 8: To check the elasticsearch is running or not.
Check the listening port with 9200
Step 9: Then check the memory lock to ensure that
mlockall is enabled,
$ sudo curl -XGET '192.168.0.34:9200/_nodes?filter_path=**.mlockall&pretty'
Result: Check
in the output that mentions ""mlockall" : true
$ sudo curl -XGET '192.168.0.34:9200/?pretty'
Check
the tagline in output. That should be
"tagline" : "You Know, for Search"
Now,
browse the elasticsearh with your localhost with port 9200. That will be "https://localhost:9200"
Installing
Kibana with Nginx:
Step 10: Download the Kibana RPM file
$ sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-x86_64.rpm
Step 11: Install the RPM downloaded rpm file.
$ sudo rpm
-ivh kibana-7.2.0-x86_64.rpm
Step 12: edit the Kibana configuration file. Enable
the below lines
$ sudo vim /etc/kibana/kibana.yml
# server.port: 5601
# server.host:
"192.168.0.34"
# elasticsearch.url:
"http://192.168.0.34:9200"
Step 13: Enable and start the Kibana
$ sudo systemctl enable kibana
$ sudo systemctl start kibana
Step 14: Check the kibana is running or not.
The Kibana
installation is finished.
Now
we need to install Nginx and configure it as reverse proxy to be able to access
Kibana from the public IP address.
Step 15: Nginx is available in the Epel repository,
install epel-release with yum.
$ sudo yum -y install epel-release
Step 16: Next, install the Nginx and httpd-tools
package.
$ sudo yum -y install nginx httpd-tools
Step 17: Edit the Nginx configuration file and remove
the 'server { }' block, so we can add a new virtual host configuration.
$ cd
/etc/nginx/
$ sudo vim nginx.conf \
Remove the server { } block. comment the Server section
Remove Server Block on
Nginx configuration
Save the file and exit vim.
Step 18: Now we need to create a new virtual host
configuration file in the conf.d directory.
Create the new file 'kibana.conf'
$ sudo vim /etc/nginx/conf.d/kibana.conf
Paste the configuration below.
server {
listen 80;
server_name <192.168.0.34 or any server name>;
auth_basic "Restricted
Access";
auth_basic_user_file
/etc/nginx/.kibana-user;
location / {
proxy_pass http://192.168.0.34:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Step 19: Then create a new basic authentication file
with the htpasswd command.
$ sudo htpasswd -c
/etc/nginx/.kibana-user admin
<TYPE YOUR PASSWORD>
Step 20: Test the Nginx configuration and make sure
there is no error. Then add Nginx to run at the boot time and start Nginx.
$ sudo nginx -t
$ sudo systemctl enable nginx
$ sudo systemctl start nginx
Fluentd Installation:
Fluentd Aggregator configuration:
Step 21: Install the td-agent
$ sudo curl -L
https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
$ sudo yum -y install gcc libcurl-devel
$ sudo yum groupinstall
"Development Tools" kernel-devel kernel-headers -y
$ sudo
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
$ sudo wget https://rubygems.org/gems/fluent-plugin-elasticsearch/versions/3.3.0
Step 22: Edit the /etc/td-agent/td-agent.conf file.
Remove the existing lines and replace with the below code.
$ sudo vim /etc/td-agent/td-agent.conf
<source>
@type forward
port 24224
</source>
<match *.log>
@type copy
<store>
@type file
path /var/log/td-agent/httpd/access_forward.log
time_slice_format %Y%m%d
time_slice_wait 10m
time_format %Y%m%dT%H%M%S%z
compress gzip
utc
</store>
<store>
@type elasticsearch_dynamic
host 192.168.0.34
port 9200
index_name fluentd-${tag_parts[1]+ "-" +
Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}
logstash_format true
time_format %Y-%m-%dT%H:%M:%S
timezone +0530
include_timestamp true
type_name fluentd
<buffer>
flush_interval 5s
flush_thread_count 3
chunk_limit_size 64m
</buffer>
</store>
</match>
Step 23: Enable and start the td-agent.service
$ sudo systemctl enable td-agent.service
$ sudo systemctl start td-agent.service
$ sudo systemctl status td-agent.service
Step 24: Check the td-agent log file.
$ sudo tail -f
/var/log/td-agent/td-agent.log
Fluentd Forwarder Configuration:
Step 25: Install the td-agent
$ sudo curl -L
https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
$ sudo yum -y install gcc libcurl-devel
$ sudo yum groupinstall
"Development Tools" kernel-devel kernel-headers -y
Step 26: Edit the Log file permissions
1.
Change
the httpd log directory permissions to “og+rx”
2.
Change
the log file permissions to “og+r” in httpd directory
Step 25: Edit the /etc/td-agent/td-agent.conf file.
Remove the existing lines and replace with the below code.
$ sudo vim /etc/td-agent/td-agent.conf
<match td.*.*>
@type tdlog
apikey YOUR_API_KEY
auto_create_table
buffer_type file
buffer_path /var/log/td-agent/buffer/td
<secondary>
@type file
path /var/log/td-agent/failed_records
</secondary>
</match>
<match debug.**>
@type stdout
</match>
<source>
@type forward
port 24224
</source>
<source>
@type http
port 8888
</source>
<source>
@type debug_agent
bind 192.168.0.22
port 24230
</source>
<source>
@type tail
path /var/log/httpd/*.log
pos_file /var/log/td-agent/access.log.pos
tag access.log
format none
time_format %Y-%m-%d %H:%M:%S,%L %z
timezone +0530
time_key time
keep_time_key true
types time:time
</source>
<match *.log>
@type copy
<store>
@type file
path /var/log/td-agent/httpd/access_forward.log
</store>
<store>
@type forward
heartbeat_type tcp
<server>
host 192.168.0.34
</server>
flush_interval 5s
</store>
</match>
Step
10: Enable and start the td-agent.service
$ sudo systemctl enable td-agent.service
$ sudo systemctl start td-agent.service
$ sudo systemctl status td-agent.service
Step 11: Check the td-agent log file.
$ sudo tail -f /var/log/td-agent/td-agent.log
