Fluentd Installation in CentOS and RHEL

Fluentd

Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.

Fluentd having two different configurations parts, which will be doing from Elasticsearch and Kibana server side that is known as Fluentd Aggregator Configuration and one more will be application side fluentd configuration which will forward the application logs from app server/webserver to Elasticstack which is known as Fluentd Forwarder Configuration. For Elasticstack configuration with Elasticsearch click here and for kibana click here.

Fluentd Aggregator configuration

Step 1:  Install the td-agent

         # curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

         #  yum -y install gcc libcurl-devel

         # yum groupinstall "Development Tools" kernel-devel kernel-headers -y

         # sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch

         # wget https://rubygems.org/gems/fluent-plugin-elasticsearch/versions/3.3.0

Step 2:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

      # vim /etc/td-agent/td-agent.conf

           <source>

             @type forward

             port 24224

          </source>

          <match *.log>

           @type copy

          <store>

           @type file

            path /var/log/td-agent/httpd/access_forward.log

            time_slice_format %Y%m%d

            time_slice_wait 10m

            time_format %Y%m%dT%H%M%S%z

            compress gzip

            utc

         </store>

         <store>

          @type elasticsearch_dynamic

           host 192.168.0.34

           port 9200

           index_name fluentd-${tag_parts[1]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}

          logstash_format true

          time_format %Y-%m-%dT%H:%M:%S

          timezone +0530

          include_timestamp true

          type_name fluentd

        <buffer>

           flush_interval 5s

           flush_thread_count 3

           chunk_limit_size 64m

        </buffer>

        </store>

        </match>

Step 3:  Enable and start the td-agent.service

        #  systemctl enable td-agent.service

        #  systemctl start td-agent.service

        #  systemctl status td-agent.service

Step 4:  Check the td-agent log file.

        # tail -f /var/log/td-agent/td-agent.log

Fluentd Forwarder Configuration:

Step 5:  Install the td-agent

         # curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

         #  yum -y install gcc libcurl-devel

         # yum groupinstall "Development Tools" kernel-devel kernel-headers -y

Step 6:  Edit the Log file permissions

               i. Change the httpd log directory permissions to “og+rx”

               ii. Change the  log file permissions to “og+r”  in httpd directory

Step 7:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

        # vim /etc/td-agent/td-agent.conf

            <match td.*.*>

             @type tdlog

             apikey YOUR_API_KEY

             auto_create_table

             buffer_type file

             buffer_path /var/log/td-agent/buffer/td

          <secondary>

           @type file

            path /var/log/td-agent/failed_records

         </secondary>

         </match>

         <match debug.**>

          @type stdout

         </match>

         <source>

          @type forward

           port 24224

         </source>

         <source>

         @type http

          port 8888

         </source>

         <source>

         @type debug_agent

          bind 192.168.0.22

          port 24230

        </source>

        <source>

         @type tail

         path /var/log/httpd/*.log

         pos_file /var/log/td-agent/access.log.pos

         tag access.log

         format none

         time_format %Y-%m-%d %H:%M:%S,%L %z

         timezone +0530

         time_key time

         keep_time_key true

         types time:time

       </source>

       <match *.log>

         @type copy

       <store>

         @type file

          path /var/log/td-agent/httpd/access_forward.log

       </store>

       <store>

        @type forward

         heartbeat_type tcp

       <server>

          host 192.168.0.34

       </server>

          flush_interval 5s

       </store>

       </match>

 Step 8:  Enable and start the td-agent.service

        #  systemctl enable td-agent.service

        #  systemctl start td-agent.service

        #  systemctl status td-agent.service

Step 11:  Check the td-agent log file.

        # tail -f /var/log/td-agent/td-agent.log

Ref: https://www.fluentd.org/

Kibana Installation in CentOS and RHEL

Kibana

Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

As we are going to create the Kibana dashboard with Nginx webserver for better security. Before setting up the Kibana, we have to install and setup the Elasticsearch in the same server. For Elasticsearch setup please click here.

Step 1:  Download the Kibana RPM file

         $ sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-x86_64.rpm

Step 2:  Install the downloaded RPM file.

         $ sudo rpm -ivh kibana-7.9.1-x86_64.rpm

Step 3:  Edit the Kibana configuration file. Enable the below lines

         $ sudo vim /etc/kibana/kibana.yml
              server.port: 5601
              server.host: "localhost"
              elasticsearch.hosts: ["http://localhost:9200"]

Step 4:  Enable and start the Kibana

         $  sudo systemctl enable kibana
          $  sudo systemctl start kibana

Step 5:  Check the kibana is running or not by checking the listening port in below command 
        

          $ netstat -lntp

NGINX Installation with Kibana 

Step 6:  Nginx is available in the Epel repository, install epel-release with yum

         $ sudo yum -y install epel-release   

Step 7: Next, install the Nginx and httpd-tools package

         $ sudo yum -y install nginx httpd-tools  

Step 8:  Edit the Nginx configuration file and remove the 'server { }' block, so we can add a new virtual host configuration.

Step 9: Now we need to create a new virtual host configuration file in the conf.d directory.   Create the new file 'kibana.conf'

        $ sudo  vim /etc/nginx/conf.d/kibana.conf
               server {
                      listen 80;
                      proxy_connect_timeout       600;
                      proxy_send_timeout          600;
                      proxy_read_timeout          600;
                      send_timeout                600;
                      index index.php index.html index.htm;
                      server_name localhost or ip;
                      auth_basic "Restricted Access";
                      auth_basic_user_file /etc/nginx/htpasswd.users;
               location / {
                      proxy_pass http://localhost:5601;
                      proxy_http_version 1.1;
                      proxy_set_header Upgrade $http_upgrade;
                      proxy_set_header Connection 'upgrade';
                      proxy_set_header Host $host;
                      proxy_cache_bypass $http_upgrade;
                      }
              location /nested {
                     alias /var/www/html;
                     #try_files $uri $uri/ @nested;      
                      }
            }

Step 10: Then create a new basic authentication file with the htpasswd command.

         $ sudo htpasswd -c /etc/nginx/htpasswd-user kibanaadmin
                    12345678

Step 11: Test the Nginx configuration and make sure there is no error. Then add Nginx to run at the boot time and start Nginx.

         $ sudo nginx -t

          $ sudo systemctl enable nginx

          $ sudo systemctl start nginx

EFK(Elasticsearch, Fluentd, Kibana) installation in CentOS machine

Elasticsearch:

               Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine tuned relevancy, and powerful analytics that scale with ease. 

Kibana:

                 Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Fluentd:

             Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language. Fluentd having two different configurations parts, which will be doing from Elasticsearch and Kibana server side that is known as Fluentd Aggregator Configuration and one more will be application side fluentd configuration which will forward the application logs from app server/webserver to Elasticstack which is known as Fluentd Forwarder Configuration.

Pre-requisites:

               System requirements,  two virtual machines

                   

                  1. elasticsearch, kibana, fluentd aggregator (one system) :192.168.0.34

                  2. In Application Node need to install fluentd forwarder  : 192.168.0.22

ElasticSearch Installation:

Step 1:  Before installing Elasticsearch, add the elastic.co key to the server.

$ sudo  rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Step 2: Now I am downloading the latest rpm of Elastic Search

$ sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-x86_64.rpm

Step 3:  Install the Downloaded RPM.

$ sudo rpm -ivh elasticsearch-7.2.0-x86_64.rpm

Step 4:  Now go to the configuration directory and edit the elasticsearch.yml configuration file. Enable the below lines in configuration file

$ sudo cd /etc/elasticsearch/

$ sudo vim elasticsearch.yml

        # bootstrap.memory_lock: true

        # network.host: 192.168.0.34

        # http.port: 9200

Step 5:  Now edit the elasticsearch.service file for the memory lock configuration. Uncomment LimitMEMLOCK line or if it is not there please add this line to limit session

$ sudo vim /usr/lib/systemd/system/elasticsearch.service

          # LimitMEMLOCK=infinity

Step 6:  Edit the sysconfig configuration file for Elasticsearch.  Uncomment line 60 and make sure the value is 'unlimited'.

$ sudo vim /etc/sysconfig/elasticsearch

            # MAX_LOCKED_MEMORY=unlimited     

The Elasticsearch configuration is finished.

Step 7:  Reload systemd, enable Elasticsearch to start at boot time, then start the service.

$ sudo systemctl daemon-reload

$ sudo systemctl enable elasticsearch

$ sudo systemctl start elasticsearch

Step 8:  To check the elasticsearch is running or not. Check the listening port with 9200

$ sudo  netstat -lntpu

Step 9:  Then check the memory lock to ensure that mlockall is enabled,

$ sudo curl -XGET '192.168.0.34:9200/_nodes?filter_path=**.mlockall&pretty'

           Result:     Check in the output that mentions ""mlockall" : true

$ sudo  curl -XGET '192.168.0.34:9200/?pretty'

               Check the tagline in output. That should be  "tagline" : "You Know, for Search"

Now, browse the elasticsearh with your localhost with port 9200. That will be "https://localhost:9200"

Installing Kibana with Nginx:

Step 10:  Download the Kibana RPM file

$ sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-x86_64.rpm

Step 11:  Install the RPM downloaded rpm file.

$ sudo rpm -ivh kibana-7.2.0-x86_64.rpm

Step 12:  edit the Kibana configuration file. Enable the below lines

$ sudo vim /etc/kibana/kibana.yml

         # server.port: 5601

         # server.host: "192.168.0.34"

         # elasticsearch.url: "http://192.168.0.34:9200"

Step 13:  Enable and start the Kibana

$ sudo systemctl enable kibana

$ sudo systemctl start kibana

Step 14:  Check the kibana is running or not.

$ sudo netstat -lntp

The Kibana installation is finished.

Now we need to install Nginx and configure it as reverse proxy to be able to access Kibana from the public IP address.

Step 15:  Nginx is available in the Epel repository, install epel-release with yum.

$ sudo yum -y install epel-release 

Step 16:  Next, install the Nginx and httpd-tools package.

$ sudo yum -y install nginx httpd-tools

Step 17:  Edit the Nginx configuration file and remove the 'server { }' block, so we can add a new virtual host configuration.

$  cd /etc/nginx/

$ sudo vim nginx.conf \         

                Remove the server { } block. comment the Server section

                    Remove Server Block on Nginx configuration

           Save the file and exit vim.

Step 18:  Now we need to create a new virtual host configuration file in the conf.d directory.   Create the new file 'kibana.conf'

$ sudo vim /etc/nginx/conf.d/kibana.conf

   Paste the configuration below.

     server {

         listen 80;

              server_name <192.168.0.34 or any server name>;

             auth_basic "Restricted Access";

             auth_basic_user_file /etc/nginx/.kibana-user;

    location / {

        proxy_pass http://192.168.0.34:5601;

        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;

        proxy_set_header Connection 'upgrade';

        proxy_set_header Host $host;

        proxy_cache_bypass $http_upgrade;

    }

}

Step 19:  Then create a new basic authentication file with the htpasswd command.

$ sudo htpasswd -c /etc/nginx/.kibana-user admin    

                   <TYPE YOUR PASSWORD>

Step 20:  Test the Nginx configuration and make sure there is no error. Then add Nginx to run at the boot time and start Nginx.

$ sudo nginx -t

$ sudo systemctl enable nginx

$ sudo systemctl start nginx

Fluentd Installation:

Fluentd Aggregator configuration:

Step 21:  Install the td-agent

$ sudo curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

$ sudo yum -y install gcc libcurl-devel

$ sudo yum groupinstall "Development Tools" kernel-devel kernel-headers -y

$ sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch

$ sudo wget https://rubygems.org/gems/fluent-plugin-elasticsearch/versions/3.3.0

Step 22:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

$ sudo vim /etc/td-agent/td-agent.conf

<source>

  @type forward

   port 24224

</source>

<match *.log>

  @type copy

    <store>

    @type file

    path /var/log/td-agent/httpd/access_forward.log

    time_slice_format %Y%m%d

    time_slice_wait 10m

    time_format %Y%m%dT%H%M%S%z

    compress gzip

    utc

  </store>

  <store>

    @type elasticsearch_dynamic

    host 192.168.0.34

    port 9200

    index_name fluentd-${tag_parts[1]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}

    logstash_format true

    time_format %Y-%m-%dT%H:%M:%S

    timezone +0530

    include_timestamp true

    type_name fluentd

    <buffer>

    flush_interval 5s

    flush_thread_count 3

    chunk_limit_size 64m

    </buffer>

  </store>

</match>

Step 23:  Enable and start the td-agent.service

$ sudo  systemctl enable td-agent.service

$ sudo systemctl start td-agent.service

$ sudo  systemctl status td-agent.service

Step 24:  Check the td-agent log file.

$ sudo tail -f /var/log/td-agent/td-agent.log

Fluentd Forwarder Configuration:

Step 25:  Install the td-agent

$ sudo curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

$ sudo yum -y install gcc libcurl-devel

$ sudo yum groupinstall "Development Tools" kernel-devel kernel-headers -y

Step 26:  Edit the Log file permissions

1.     Change the httpd log directory permissions to “og+rx”

2.     Change the  log file permissions to “og+r”  in httpd directory

Step 25:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

$ sudo vim /etc/td-agent/td-agent.conf

<match td.*.*>

  @type tdlog

  apikey YOUR_API_KEY

  auto_create_table

  buffer_type file

  buffer_path /var/log/td-agent/buffer/td

  <secondary>

    @type file

    path /var/log/td-agent/failed_records

  </secondary>

</match>

<match debug.**>

  @type stdout

</match>

<source>

  @type forward

  port 24224

</source>

<source>

  @type http

  port 8888

</source>

<source>

  @type debug_agent

  bind 192.168.0.22

  port 24230

</source>

<source>

  @type tail

  path /var/log/httpd/*.log

  pos_file /var/log/td-agent/access.log.pos

  tag access.log

  format none

  time_format %Y-%m-%d %H:%M:%S,%L %z

  timezone +0530

  time_key time

  keep_time_key true

  types time:time

</source>

<match *.log>

   @type copy

   <store>

    @type file

    path /var/log/td-agent/httpd/access_forward.log

  </store>

  <store>

    @type forward

    heartbeat_type tcp

    <server>

    host 192.168.0.34

    </server>

    flush_interval 5s

  </store>

</match>

 Step 10:  Enable and start the td-agent.service

$ sudo systemctl enable td-agent.service

$ sudo systemctl start td-agent.service

$ sudo systemctl status td-agent.service

Step 11:  Check the td-agent log file.

$ sudo tail -f /var/log/td-agent/td-agent.log