Kuberenetes Introduction - Part 1

 

What is Kubernetes?

Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.

Kubernetes helps to package and distribute apps using image format and container technologies.

It groups containers that make up an application into logical units for easy management and discovery.

Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

Kubernetes Features

Automatic bin packing

• Automatically places containers based on their resource requirements and other  constraints, while not sacrificing availability.

• Mix critical and best-effort workloads in order to drive up utilization and save even more resources.

Automated rollouts and rollbacks

• Kubernetes progressively rolls out changes to your application or its configuration,  while monitoring application health to ensure it doesn't kill all your instances at the same time.

• If something goes wrong, Kubernetes will roll back the change for you.

• Take advantage of a growing ecosystem of deployment solutions.

Horizontal scaling

• Scale your application up and down with a simple command, with a UI, or automatically based on CPU usage.

Service discovery and load balancing

• No need to modify your application to use an unfamiliar service discovery mechanism.

• Kubernetes gives containers their own IP addresses and a single DNS name for a set of containers and can load-balance across them.

Storage orchestration

• Automatically mount the storage system of your choice, whether from local storage, a  public cloud provider such as GCP or AWS, or a network storage system such as NFS,  iSCSI, Gluster, Ceph, Cinder, or Flocker.

Self-healing

• Restarts containers that fail, replace and reschedule containers when nodes die, kills  containers that don't respond to your user-defined health check, and doesn't advertise  them to clients until they are ready to serve.

Secret and configuration management

• Deploy and update secrets and application configuration without rebuilding your image and without exposing secrets in your stack configuration.

Kubernetes Architecture

Node Controller/ Master

• The node controller is a Kubernetes master component that manages various aspects of nodes.

• The node controller has multiple roles in a node’s life. (1). The first is assigning a CIDR block to the node when it is registered (if the CIDR  assignment is turned on).  (2). The second is keeping the node controller’s internal list of nodes up to date with the cloud provider’s list of available machines. (3).  The third is monitoring the nodes’ health.

• The services on a Node Controller/Master Node include kube-api, kube-scheduler, kube-controller-manager and etcd.

Node/Minions

• A node is a worker machine in Kubernetes, previously known as a minion.

• A node may be a VM or physical machine, depending on the cluster.

• Each node has the services necessary to run pods and is managed by the node components.

• The services on a Node include Docker, kubelet and kube-proxy, CNI.

Node Controller/Master Components

Kube-ApiServer

• The Kubernetes API-server generally validates the configuration data store in ‘Etcd’ and the details of the deployed container that are in agreement.

• It also provides a RESTful interface to make communication easy.

Kube-Controller-Manager

• It is generally responsible for handling the cluster level function such as replication controller.

• Whenever the desired state of the cluster changes it is written to Etcd and then the controller manager tries to bring up the cluster in the desired state.

Kube-Schedule Server

• It is responsible for assigning tasks to nodes/minions in the cluster.

Etcd

• It is an open-source key-value store developed by the CoreOs team. Kubernetes uses  ‘Etcd’ to store the configuration data accessed by all nodes (minions and master) in the cluster.

Node Components

Kubelet

• Host-level pod management.

• It deals with pod specifications that are defined in YAML or JSON format.

• It is responsible for managing pods and their containers.

• It is an agent process that runs on each node.

• Kubelet takes the pod specifications and checks whether the pods are running healthy or not.

Kube-Proxy

• Every node in the cluster runs a simple network proxy. Using a proxy node in cluster routes requests to the correct container in a node.

• Manages the container network (IP addresses and ports) based on the network service manifests received  from the Kubernetes master.

Docker

• An API and framework built around Linux Containers (LXC) that allows for the easy management of containers and their images.

CNI

• A network overlay that will allow containers to communicate across multiple hosts.

Setting Up Azure CLI on Linux for Seamless Azure Management

Introduction

The Azure CLI (Command-Line Interface) is a powerful tool that enables seamless interaction with Azure resources. In this blog post, we'll walk through the process of setting up Azure CLI on a Linux machine. Whether you are an administrator, developer, or an Azure enthusiast, having Azure CLI locally on your Linux computer allows you to execute administrative commands efficiently.

Prerequisites

Before we start, ensure you have the following prerequisites:

• A Linux computer
• Administrative privileges on the Linux machine

Step-by-Step Guide

Step 1: Import Microsoft Repository Key

Open your terminal and run the following command to import the Microsoft repository key:

bash
$ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc

Step 2: Create Local Azure-CLI Repository Information

Create and configure the local Azure CLI repository information by running the following commands:

bash
  $ sudo sh -c 'echo -e "[azure-cli]
  name=Azure CLI
  baseurl=https://packages.microsoft.com/yumrepos/azure-cli
  enabled=1
  gpgcheck=1
  gpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'

Step 3: Install Azure CLI

Install Azure CLI using the yum install command:

bash
$ sudo yum install azure-cli -y

Step 4: Check the Installed Version

Verify the successful installation by checking the installed version of Azure CLI:

bash
$ az --version

Conclusion

Congratulations! You've successfully set up Azure CLI on your Linux machine. With Azure CLI, you can now execute a wide range of administrative commands, manage Azure resources, and streamline your interactions with Azure services directly from the command line.

For more details and additional options, you can refer to the official documentation: Install Azure CLI on Linux.

Now you're ready to explore and harness the power of Azure CLI for efficient Azure management on your Linux environment. Happy coding!

Azure Cli Installation in Ubuntu (Debian)

The Azure CLI is a cross-platform command-line tool that can be installed locally on Linux computers. You can use the Azure CLI on Linux to connect to Azure and execute administrative commands on Azure resources. The CLI on Linux allows the execution of various commands through a terminal using interactive command-line prompts or a script.

Step 1:  Get packages needed for the install process

   $ sudo apt-get update
   $ sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg
   
Step 2: Download and install the Microsoft signing key

$ sudo curl -sL https://packages.microsoft.com/keys/microsoft.asc |
    gpg --dearmor |
    sudo tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null
    
Step 3: Add the Azure CLI software repository

$ sudo AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" |
    sudo tee /etc/apt/sources.list.d/azure-cli.list

Step 4: Update repository information and install the azure-cli package

$ sudo apt-get update
$ sudo apt-get install azure-cli

Step 5: Check the Azure cli version installed

$ az --version

Ref: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots= apt

Logstash Installation and configuration in CentOS

Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination. It is most often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

Stages:

 1. Install Logstash  & Configuring Logstash

 2. Install Filebeat (on the Client Servers) & Configuring Filebeat

Stage 1:

Install Logstash

$ sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.6.0.rpm

$ sudo rpm –ivh logstash-7.6.0.rpm

Configuring Logstash

As part of configuring the Logstash , we need to generate the Selfsigned certs. Basically we can generate certs in two methods i). Based on Private IP ii). Based on Domain.

Option 1: Based on Private IP

Add a SSL certificate based on the IP address of the ELK server. Add the ELK server’s private IP in /etc/pki/tls/openssl.cnf.

$ sudo vi /etc/pki/tls/openssl.cnf

Add the following line with private Ip just below [ v3_ca ] section

subjectAltName = IP: 192.168.20.158

 GENERATE A SELF-SIGNED CERTIFICATE VALID FOR 365 DAYS

$ cd /etc/pki/tls

$ sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/ssl-logstash-forwarder.key -out certs/ssl-logstash-forwarder.crt

Option 2 – Based on domain (FQDN)

$ cd /etc/pki/tls

$ sudo openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/ssl-logstash-forwarder.key -out certs/ssl-logstash-forwarder.crt

LOGSTASH INPUT, FILTER, OUTPUT FILES

Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs – usually specified as separate files.

INPUT.CONF 

Insert the following lines into it. This is necessary for Logstash to “learn” how to process beats coming from clients. Make sure the path to the certificate and key match the right paths as outlined in the previous step:  This specifies that Logstash will listen on tcp port 5044 i.e. log-forwarder will connect at this port to send logs.

$ sudo vi /etc/logstash/conf.d/01-beats-input.conf

input {

beats {

port => 5044

ssl => true

ssl_certificate => "/etc/pki/tls/certs/ssl-logstash-forwarder.crt"

ssl_key => "/etc/pki/tls/private/ssl-logstash-forwarder.key"

}

}

FILTER.CONF

This filter looks for logs that are labeled as “syslog” type (by Filebeat), and it will try to use grok to parse incoming syslog logs to make it structured and query-able.

$ sudo vi /etc/logstash/conf.d/01-beats-filter.conf

filter {

if [type] == "syslog" {

grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ]

}

syslog_pri { }

date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

}

}

OUTPUT.CONF

In output.conf basically we configured for Logstash to store the beats data in Elasticsearch which is running at localhost:9200, in an index named after the beat used (filebeat, in our case).

$ sudo vi /etc/logstash/conf.d/01-beats-output.conf

output {

elasticsearch {

hosts => ["localhost:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}

}

START AND ENABLE LOGSTASH

$ sudo systemctl daemon-reload

$ sudo systemctl start logstash

$ sudo systemctl enable logstash

Stage 2:

Install Filebeat (on the Client Servers)

We need to install the Filebeat in Client servers(means from which server we are planning to get the logs) can be one or more servers, the below process need to follow for all client servers. 

COPY THE SSL CERTIFICATE FROM THE ELK SERVER TO THE CLIENT(S)

$ sudo scp /etc/pki/tls/certs/logstash-forwarder.crt root@<client server IP>:/etc/pki/tls/certs/

NOTE

Once copied the generated certs to client servers, as a next we need to Perform the following steps on the client machine (ones sending logs to ELK server)

IMPORT THE ELASTICSEARCH PUBLIC GPG KEY

$ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

CREATE AND EDIT A NEW YUM REPOSITORY FILE FOR FILEBEAT

$ sudo vi /etc/yum.repos.d/filebeat.repo

Add the following repository configuration to filebeat.repo file

[elastic-7.x]

name=Elastic repository for 7.x packages

baseurl=https://artifacts.elastic.co/packages/7.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

INSTALL THE FILEBEAT PACKAGE

$ sudo yum -y  install filebeat

CONFIGURE FILEBEAT

Edit Filebeat configuration file & Replace ‘elk_server_private_ip‘ with the private IP of your ELK server i.e. hosts: [“IP:5044”]

$ sudo vi /etc/filebeat/filebeat.yml

filebeat:

prospectors:

-

paths:

- /var/log/secure

- /var/log/messages

#  - /var/log/*.loginput_type: logdocument_type: syslogregistry_file: /var/lib/filebeat/registryoutput:

logstash:

hosts: ["elk_server_private_ip:5044"]

bulk_max_size: 1024tls:

certificate_authorities: ["/etc/pki/tls/certs/ssl-logstash-forwarder.crt"]shipper:logging:

files:

rotateeverybytes: 10485760 # = 10MB

START AND ENABLE FILEBEAT

$ systemctl start filebeat

$ systemctl enable filebeat

Test Filebeat

On your ELK Server, verify that Elasticsearch is receiving the data by querying for the Filebeat index with this command:

$ curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

Ref: https://www.elastic.co/logstash/

Ref: https://www.elastic.co/beats/filebeat