Jenkins Instllation using Binary and Distribution methods in Ubuntu

Jenkins is an Open Source automation server. It helps to automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery. It is a server based system that runs in servlet containers such as Apache Tomcat ...

PreRequisites:

To setup the Jenkins as a prerequisite we need java need to install in the server and then setup the Jenkins.

      Step 1: Setting up the openjdk repo in machine

         $ cd /opt
         $ sudo add-apt-repository ppa:openjdk-r/ppa

      Step 2: update the OS after adding repo and Install jdk 8
         

         $ sudo apt update
         $ sudo apt install openjdk-8-jdk

Jenkins installation using Binary:

     Step 3:  Download and install the necessary GPG key with the command

         $ sudo wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -

     Step 4: Add the necessary repository with the command

         $  sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'  

    Step 5: Update apt with the command 

        $ sudo apt-get update

    Step 6: Install Jenkins with the command 

        $ sudo apt-get install jenkins -y

    Step 7:  Start the Jenkins Service

        $ sudo systemctl start jenkins

Jenkins Installation using Distribution method

Tomcat Installation:    from https://tomcat.apache.org/download-90.cgi

    Step 1: Install the tomcat from above URL

        $ cd /opt/

        $ sudo wget https://mirrors.estointernet.in/apache/tomcat/tomcat-9/v9.0.37/bin/apache-tomcat-9.0.37.tar.gz

    Step 2: Un tar the downloaded file and change the directory name

        $ tar -xvzf apache-tomcat-9.0.37.tar.gz -C /opt

        $ mv /opt/apache-tomcat-9.0.37/ /opt/tomcat

        $ cd tomcat/bin

    Step 3: Change the mode od directory, give the execution permission for user in current bin directory.

        $ sudo chmod u+x *  

Jenins Installation:   from https://updates.jenkins-ci.org/

    Step 4: Install the jenkins.war file from above url

        $ sudo wget http://mirrors.jenkins.io/war-stable/latest/jenkins.war

    Step 5: Move the Jenkins.war file to /opt/tomcat/webapps

        $ sudo mv /opt/jenkins.war /opt/tomcat/webapps/

        $ sudo apt update

        $ cd /opt/tomcat/bin

    Step 6: Start the jenkins from the current directory (/opt/tomcat/bin) 

        $ sudo sh startup.sh

Fluentd Installation in CentOS and RHEL

Fluentd

Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.

Fluentd having two different configurations parts, which will be doing from Elasticsearch and Kibana server side that is known as Fluentd Aggregator Configuration and one more will be application side fluentd configuration which will forward the application logs from app server/webserver to Elasticstack which is known as Fluentd Forwarder Configuration. For Elasticstack configuration with Elasticsearch click here and for kibana click here.

Fluentd Aggregator configuration

Step 1:  Install the td-agent

         # curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

         #  yum -y install gcc libcurl-devel

         # yum groupinstall "Development Tools" kernel-devel kernel-headers -y

         # sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch

         # wget https://rubygems.org/gems/fluent-plugin-elasticsearch/versions/3.3.0

Step 2:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

      # vim /etc/td-agent/td-agent.conf

           <source>

             @type forward

             port 24224

          </source>

          <match *.log>

           @type copy

          <store>

           @type file

            path /var/log/td-agent/httpd/access_forward.log

            time_slice_format %Y%m%d

            time_slice_wait 10m

            time_format %Y%m%dT%H%M%S%z

            compress gzip

            utc

         </store>

         <store>

          @type elasticsearch_dynamic

           host 192.168.0.34

           port 9200

           index_name fluentd-${tag_parts[1]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}

          logstash_format true

          time_format %Y-%m-%dT%H:%M:%S

          timezone +0530

          include_timestamp true

          type_name fluentd

        <buffer>

           flush_interval 5s

           flush_thread_count 3

           chunk_limit_size 64m

        </buffer>

        </store>

        </match>

Step 3:  Enable and start the td-agent.service

        #  systemctl enable td-agent.service

        #  systemctl start td-agent.service

        #  systemctl status td-agent.service

Step 4:  Check the td-agent log file.

        # tail -f /var/log/td-agent/td-agent.log

Fluentd Forwarder Configuration:

Step 5:  Install the td-agent

         # curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh

         #  yum -y install gcc libcurl-devel

         # yum groupinstall "Development Tools" kernel-devel kernel-headers -y

Step 6:  Edit the Log file permissions

               i. Change the httpd log directory permissions to “og+rx”

               ii. Change the  log file permissions to “og+r”  in httpd directory

Step 7:  Edit the /etc/td-agent/td-agent.conf file. Remove the existing lines and replace with the below code.

        # vim /etc/td-agent/td-agent.conf

            <match td.*.*>

             @type tdlog

             apikey YOUR_API_KEY

             auto_create_table

             buffer_type file

             buffer_path /var/log/td-agent/buffer/td

          <secondary>

           @type file

            path /var/log/td-agent/failed_records

         </secondary>

         </match>

         <match debug.**>

          @type stdout

         </match>

         <source>

          @type forward

           port 24224

         </source>

         <source>

         @type http

          port 8888

         </source>

         <source>

         @type debug_agent

          bind 192.168.0.22

          port 24230

        </source>

        <source>

         @type tail

         path /var/log/httpd/*.log

         pos_file /var/log/td-agent/access.log.pos

         tag access.log

         format none

         time_format %Y-%m-%d %H:%M:%S,%L %z

         timezone +0530

         time_key time

         keep_time_key true

         types time:time

       </source>

       <match *.log>

         @type copy

       <store>

         @type file

          path /var/log/td-agent/httpd/access_forward.log

       </store>

       <store>

        @type forward

         heartbeat_type tcp

       <server>

          host 192.168.0.34

       </server>

          flush_interval 5s

       </store>

       </match>

 Step 8:  Enable and start the td-agent.service

        #  systemctl enable td-agent.service

        #  systemctl start td-agent.service

        #  systemctl status td-agent.service

Step 11:  Check the td-agent log file.

        # tail -f /var/log/td-agent/td-agent.log

Ref: https://www.fluentd.org/

Kibana Installation in CentOS and RHEL

Kibana

Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

As we are going to create the Kibana dashboard with Nginx webserver for better security. Before setting up the Kibana, we have to install and setup the Elasticsearch in the same server. For Elasticsearch setup please click here.

Step 1:  Download the Kibana RPM file

         $ sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-x86_64.rpm

Step 2:  Install the downloaded RPM file.

         $ sudo rpm -ivh kibana-7.9.1-x86_64.rpm

Step 3:  Edit the Kibana configuration file. Enable the below lines

         $ sudo vim /etc/kibana/kibana.yml
              server.port: 5601
              server.host: "localhost"
              elasticsearch.hosts: ["http://localhost:9200"]

Step 4:  Enable and start the Kibana

         $  sudo systemctl enable kibana
          $  sudo systemctl start kibana

Step 5:  Check the kibana is running or not by checking the listening port in below command 
        

          $ netstat -lntp

NGINX Installation with Kibana 

Step 6:  Nginx is available in the Epel repository, install epel-release with yum

         $ sudo yum -y install epel-release   

Step 7: Next, install the Nginx and httpd-tools package

         $ sudo yum -y install nginx httpd-tools  

Step 8:  Edit the Nginx configuration file and remove the 'server { }' block, so we can add a new virtual host configuration.

Step 9: Now we need to create a new virtual host configuration file in the conf.d directory.   Create the new file 'kibana.conf'

        $ sudo  vim /etc/nginx/conf.d/kibana.conf
               server {
                      listen 80;
                      proxy_connect_timeout       600;
                      proxy_send_timeout          600;
                      proxy_read_timeout          600;
                      send_timeout                600;
                      index index.php index.html index.htm;
                      server_name localhost or ip;
                      auth_basic "Restricted Access";
                      auth_basic_user_file /etc/nginx/htpasswd.users;
               location / {
                      proxy_pass http://localhost:5601;
                      proxy_http_version 1.1;
                      proxy_set_header Upgrade $http_upgrade;
                      proxy_set_header Connection 'upgrade';
                      proxy_set_header Host $host;
                      proxy_cache_bypass $http_upgrade;
                      }
              location /nested {
                     alias /var/www/html;
                     #try_files $uri $uri/ @nested;      
                      }
            }

Step 10: Then create a new basic authentication file with the htpasswd command.

         $ sudo htpasswd -c /etc/nginx/htpasswd-user kibanaadmin
                    12345678

Step 11: Test the Nginx configuration and make sure there is no error. Then add Nginx to run at the boot time and start Nginx.

         $ sudo nginx -t

          $ sudo systemctl enable nginx

          $ sudo systemctl start nginx

ElasticSearch Installation in CentOS and RHEL

#Elasticsearch  #Installation #CentOS #RHEL

What is Elasticsearch?

      Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine tuned relevancy, and powerful analytics that scale with ease. 

Now, we will see how to install the Elasticsearch in CentOS and RHEL servers. Basically, we have to install JAVA has to be prerequisite for Elasticsearch. So first we will install Java and setup the environment variables and then install the Elasticsearch with respect to basic production configuration.

Prerequisite:

To setup the Elasticsearch we require Java need to installed in Elasticsearch stack, using following steps we will be able to install the java.

Step 1:  Installing the openjdk 1.8 in the elasticsearch server

    

     a) update the server and install java with below command.

          $ sudo yum update 
          $ sudo yum install java-1.8.0-openjdk -y

     b) Once installed the java, check the java version using below command

          $ sudo java -version

     c) After validating the java version , configure the java home to use in run time in .bashrc file

          $ sudo update-alternatives --config java

          $ sudo vi .bashrc
           export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java

 

     d) Execute the .bashrc to enable the java home 

          $ source .bashrc

Elasticsearch Setup:

Once the Java setup is completed, we need to setup the elasticsearch using below steps

Step 2: Before installing Elasticsearch, add the elastic.co key to the server.

         $ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Step 3: Now, Download the Latest RPM of Elasticsearch

        $ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.1-x86_64.rpm

Step 4: Install the Downloaded RPM

       $ rpm -ivh elasticsearch-7.9.1-x86_64.rpm

Step 5:  Now go to the configuration directory and edit the elasticsearch.yml configuration file. Enable the below lines in configuration file

      $ cd /etc/elasticsearch/
      $ vim elasticsearch.yml
             bootstrap.memory_lock: true
             xpack.monitoring.enabled: true
             network.host: localhost
             http.port: 9200
             http.max_content_length: 1.6gb

     And Save the file.

Step 6: Now edit the elasticsearch.service file for the memory lock configuration. Uncomment LimitMEMLOCK line or if it is not there please add this line to limit session

     $ vim /usr/lib/systemd/system/elasticsearch.service

           LimitMEMLOCK=infinity

Step 7: Edit the sysconfig configuration file for Elasticsearch.  Uncomment line 60 and make sure the value is 'unlimited'.

     $ vim /etc/sysconfig/elasticsearch
               MAX_LOCKED_MEMORY=unlimited    
       MAX_OPEN_FILES=65535
               MAX_MAP_COUNT=262144

Step 8: Reload systemd, enable Elasticsearch to start at boot time, then start the service.

     $ sudo systemctl daemon-reload
     $ sudo systemctl enable elasticsearch
     $ sudo systemctl start elasticsearch

Step 9: To check the elasticsearch is running or not. Check the listening port with 9200

     $ netstat -lntp 

Step 10: Then check the memory lock to ensure that mlockall is enabled,

     $ curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'
     $ curl -XGET 'localhost:9200/?pretty'

                                                                                                            Next: Kibana Installation in CentOS

Docker Introduction

Docker Introduction

• Docker is an Open Source Platform designed to build, distribute and run applications inside the container.
• Docker is Virtual box that packages applications.
• Docker implements a high level API to proceed the light weight containers that run processors.
• Also Docker is a tool for "Rapid Application Development"
• Docker will use in Dev, Testing, Stage but before will not use in production because of security reason.
• But now Docker itself has clearances, They integrated with SE-Linux(know that is for Security purpose).

Virtual Machines Vs Containers

                            VM's                                                          Docker Containers
   

                 -> More time to Create                               -> Less time to create

                 -> Dedicated OS                                        -> Common OS

                 -> Specific bin/lib                                       -> Common bin/lib

                 -> Dedicated Resources                              -> Common resources

                 -> Memory wastage                                    -> No Memory Wastage

                 -> Low Performance                                   -> high performance

                 -> Complex Configure                                -> Simple Configuration

                 -> Heavy Weight (Gb's)                              -> Light Memory (Mb's)

                 -> Memory can not share                             -> Can Share Memory

Docker Core Terminology

Docker mainly have Core parts. Those are

                   ** Images
                   ** Containers
                   ** Dockerfile
                   ** Docker Volumes
                   ** Docker Compose
                   ** Docker Network

Docker Images: 

• Docker image is a collection of files and source meta data.
• Images are made of layers, Conceptually stacked on top of each other.
• Images can share layers to optimize the disk usage, transfer times and memory usage.
• Docker image is Read Only file system.

Docker Containers:

• A container is an executable form of docker image.
• A Container is an encapsulated set of processes running in read-write copy of the file system.

Dockerfile:

• Creates images automatically by using a build script: <Dockerfile>
• Can be versionized in a version control system like Git or SVN along with all dependencies.
• Docker hub can automatically build images based on dockerfiles on GitHub.

Docker Volumes:

• In Docker, we have a separate volume that can shared across containers. These are known as Data Volumes.
• They are initialized when the container is created.
• They can be shared and also reused among many containers.
• Any changes to the volume itself can be made directly.
• They exist even after the container is deleted.

          Ex: # docker run -it -v /root/files:/files ubuntu

Docker Compose:

• Docker Compose is a tool for defining and running multi-container Docker applications.
• With Compose, we use a compose file to configure application's services.
• Then, by using a single command we can create and start all the services from the given configuration requirement.

Docker Network: 

• To communicate Docker Containers with each other and the outside world via the host machine, there has to be a layer of networking involved.
• Docker supports different types of networks, each fit for certain use cases.
• Different Network Types    

       Docker comes with network drivers geared towards different use cases. The most common network types being:

• Bridge
• Overlay
• Macvlan

Run Platforms:

• Various Linux Distributions (Ubuntu, CentOS, Fedora, RHEL, OpenSUSE, ...)
• Cloud (AWS, GCP, Rackspace)
• 2014-10: Microsoft announces plans to integrate Docker with next release of Windows Server.
• 2013-03: Releases as Open Source Tool.

Install Grafana in CentOS with YUM/Distribution-method/RPM Methods

Grafana Introduction

• Grafana is a  multi-platform open source analytics and interactive visualisation software
• It is available since 2014
• It provides charts, graphs and alerts for the web when connected to supported data sources.
• It is expandable through a plug-in system.

Supported operating systems

• Debian/Ubuntu
• RPM-Based Linux (CentOS, Fedora, Opensuse, RHEL)
• MacOS
• Windows

Installation of Grafana on other operating systems is possible, but not supported.

Hardware Recommendations

• Grafana does not use a lot of resources and is very lightweight in use of memory and CPU.
• Minimum recommended memory: 255 MB 
• Minimum recommended CPU: 1
• Some features might require more memory or CPUs. Features require more resources include:

               a. Server side rendering of images

               b. Alerting

               c.  Data source proxy

Installation with YUM repo :

Step1 : Updating the server

$ sudo yum update

Step 2: Adding grafana repo for installing grafana

$ sudo vi /etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Step 3: Installing Grafana

$ sudo yum install grafana


Step 4: Start and Enable the grafana service

$ sudo systemctl daemon-reload
$ sudo systemctl start grafana-server
$ sudo systemctl status grafana-server
$ sudo systemctl enable grafana-server

Installation with Distribution Method (Using Binary):

Step 1: Downloading Grafana Zip file
$ sudo wget https://dl.grafana.com/oss/release/grafana-6.6.0.linux-amd64.tar.gz

Step 2: Unzip the Grafana zip file

$ tar -zxvf grafana-6.6.0.linux-amd64.tar.gz

Step 3: Start and enable the grafana server

$ sudo systemctl daemon-reload
$ sudo systemctl start grafana-server
$ sudo systemctl status grafana-server
$ sudo systemctl enable grafana-server

Installation with RPM:

Step 1: Adding Grafana RPM key


 $ sudo wget https://packages.grafana.com/gpg.key

Step 2: Install some dependencies to install grafana with RPM


 $ sudo yum install initscripts urw-fonts wget

Step 3: Download the grafana RPM file

 $ sudo wget https://dl.grafana.com/oss/release/grafana-6.6.0-1.x86_64.rpm
 $ sudo rpm -Uvh grafana-6.6.0-1.x86_64.rpm

Step 4: Installing Grafana

$ sudo yum localinstall grafana-6.6.0-1.x86_64.rpm

Step 3: Start and Enable the Grafana Server

 $ sudo systemctl daemon-reload
 $ sudo systemctl start grafana-server
 $ sudo systemctl status grafana-server
 $ sudo systemctl enable grafana-server

Referral Page : https://grafana.com/docs/grafana/latest/installation/

Install Ant in Ubuntu 16.04/18.04/20.04 Machine

APACHE ANT

• ANT stands for another neat tools
• It is java based build tool from Apache Software Fundation
• Ant uses build.xml file 
• Ant was created by james duncan davidson

Ant- Build Files:

• Ants build file is called build.xml should reside in base directory(base directory means where the source code will maintained) of the project
• In build.xml or any xml file it start with an xml tag project name and target a single project can have multiple targets
• There should be no blank lines or white spaces before the xml declaration

Installation: 

    We can install ANT in different ways, those are 1. Using "apt" command module

                                                                                 2. Using "distribution method"

Here, we will see both the ways how to install ANT.

Prerequisites: 

Java should be install in the server.

Step 1: Update the Server 

$ sudo apt update -y 


Step 2: Install Java latest version, basically this command will install JAVA Runtime Environment


$ sudo apt install default-jre


Install Apache ANT in Ubuntu using "apt" command module

Step 3:  Update the Server Install Ant


$ sudo apt update 
$ sudo apt install ant -y
To verify the Ant installation
$ ant -version

Install Apache Ant Using "Distribution Method"

Step 1: Enter into the "opt" directory


$ cd /opt/



Step 2:  Download the tar or zap source file from http://ant.apache.org/bindownload.cgi


$ sudo wget http://mirrors.estointernet.in/apache//ant/binaries/apache-ant-1.10.5-bin.zip

Step 3:  Exatract the zip file or tar file


$ unzip apache-ant-1.10.5-bin.zip



Step 4: Add the ANT_HOME path.



$ sudo ANT_HOME=/opt/apache-ant-1.10.5




Step 5:  Now export the PATH



$ sudo PATH=$PATH:${ANT_HOME}/bin

Step 6: Now enter into the apache-ant-1.10.5


$ cd apache-ant-1.10.5




Step 7:  Now, Run the below command to start the "Build"


$ sudo ant -f fetch.xml -Ddest=system

NFS Server Installation

Important files for NFS Configration

• /etc/exports: Its a main configuration file of NFS, all exported files and directories are defined in this file at the NFS Server end.
• /etc/fstab: To mount a NFS directory on your system across the reboots, we need to make an entry in /etc/fstab.
• /etc/sysconfig/nfs: Configuration file of NFS to control on which port rpc and other services are listening.

To setup NFS mounts, we’ll be needing at least two Linux/Unix machines. Here in this tutorial, I’ll be using two servers.

             NFS Server: nfs.example.com with IP-192.168.0.63

             NFS Client : nfs-client.example.com with IP-192.168.0.64

At NFS Server End

Step 1: As the first step, we will install these packages on the CentOS server with yum

#      sudo yum install nfs-utils -y

Step 2:  Now create the directory that will be shared by NFS

#  sudo mkdir /var/nfsshare

Step 3: Change the permissions of the folder as follows:

# chmod -R 755 /var/nfsshare
# chown nobody:nobody /var/nfshare

NOTE: We use /var/nfsshare as shared folder, if we use another drive such as the /home directory, then the permission chnges will cause a massive permissions problem and ruin the whole hierarchy. So in case we want to share the /home directory then permissions must not be changed.

Step 4: 

#      PATH=$PATH:${ANT_HOME}/bin

How to Install ANT(Build Tool) in CentOS machine

APACHE ANT

• ANT stands for another neat tools
• It is java based build tool from Apache Software Fundation
• Ant uses build.xml file 
• Ant was created by james duncan davidson

Ant- Build Files:

• Ants build file is called build.xml should reside in base directory(base directory means where the source code will maintained) of the project
• In build.xml or any xml file it start with an xml tag project name and target a single project can have multiple targets
• There should be no blank lines or white spaces before the xml declaration

Ant Installation In CentOS Machine

Step 1: Update the Virtual machine

$ sudo yum update -y 


Step 2: Enter into the "opt" directory


$ cd /opt/


Step 3:  Download the tar or zap source file from http://ant.apache.org/bindownload.cgi



$ sudo wget http://mirrors.estointernet.in/apache//ant/binaries/apache-ant-1.10.5-bin.zip

Step 4:  Exatract the zip file or tar file


$ sudo unzip apache-ant-1.10.5-bin.zip


Step 5: Add the ANT_HOME path.


$ sudo ANT_HOME=/opt/apache-ant-1.10.5




Step 6:  Now export the PATH


$ sudo PATH=$PATH:${ANT_HOME}/bin

Step 7: Now enter into the apache-ant-1.10.5


$ sudo cd apache-ant-1.10.5




Step 8:  Now, Run the below command to start the "Build"


$ sudo ant -f fetch.xml -Ddest=system