Installing Logstash, Elasticsearch, Kibana (ELK stack) & Filebeat on Ubuntu 14.04


ELK_stack_1

Requirements.

Three servers:

– for AppServer (Filebeat)

– for Logstash + Kibana + Nginx

– for Elastisearch

Set all date to UTC (or local time)

# dpkg-reconfigure tzdata

# apt-get install ntp

# service ntp stop && ntpdate-debian && service ntp start

Install Java 8 on logstash & elastisearch servers

# add-apt-repository -y ppa:webupd8team/java

# apt-get update

# apt-get -y install oracle-java8-installer

On Elasticsearch server

# wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

# echo ‘deb http://packages.elastic.co/elasticsearch/2.x/debian stable main’ > /etc/apt/sources.list.d/elasticsearch-2.x.list

# apt-get update

# apt-get -y install elasticsearch

Configure:

# vim /etc/elasticsearch/elasticsearch.yml

 network.host: LOCAL_IP_ELASTIC_SERVER 

# service elasticsearch restart

Make elasticsearch bootable.

# update-rc.d elasticsearch defaults 95 10

Check:

# ss -nlp |grep 9200

On Logstash server
# wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
# echo ‘deb http://packages.elasticsearch.org/logstash/2.1/debian stable main’ > /etc/apt/sources.list.d/logstash.list
# apt-get update
# apt-get install logstash
Configure:
# vim /etc/logstash/conf.d/02-beats-input.conf

input { 
 beats { 
  port => 5044 
 } 
}

# vim /etc/logstash/conf.d/10-syslog-filter.conf

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

# vim /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["LOCAL_IP_ELASTIC_SERVER:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

# service logstash configtest

# service logstash restart

# update-rc.d logstash defaults 96 9

Check:

# ss -nlp |grep 5044

Install Kibana (Logstash server)

On second server (were will install logstash).

Create user & group kibana.

# groupadd -g 999 kibana && useradd -u 999 -g 999 kibana

# cd ~; wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz

# tar xvf kibana-*.tar.gz

Configure:

# vim ~/kibana-4*/config/kibana.yml

server.host: "localhost"
elasticsearch.url: "http://LOCAL_IP_ELASTIC_SERVER:9200"

# mkdir -p /opt/kibana && cp -R ~/kibana-4*/* /opt/kibana/ && chown -R kibana: /opt/kibana

# cd /etc/init.d && sudo curl -o kibana https://gist.githubusercontent.com/valch85/27f19cf661acc4910ad1/raw/4ac56cea127880e65035501f76419bb9758075c0/kibana-4.x-init

# cd /etc/default && sudo curl -o kibana https://gist.githubusercontent.com/valch85/bee274b89ab2614804a1/raw/9ca548abf337b2e0b50045632c4d1e42422f6fc0/kibana-4.x-default

# chmod +x /etc/init.d/kibana

# update-rc.d kibana defaults 96 9

# service kibana start

Check:

# ss -nlp |grep 5601

Install Nginx (Logstash server)

# apt-get install nginx apache2-utils

Generate credential to limit access to kibana.

# htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

# vim /etc/nginx/sites-available/default

server {
    listen 80;
    server_name EXTERNAL_IP_LOGSTASH_SERVER;
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade; 
    }
}

# service nginx restart

Check:

# ss -nlp |grep 80

App server

Install filebeat:

# wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

# echo ‘deb https://packages.elastic.co/beats/apt stable main’  > /etc/apt/sources.list.d/beats.list 

# apt-get update && apt-get install filebeat

Configure filebeat:

# vim /etc/filebeat/filebeat.yml

filebeat.yml excerpt 1 of 2

...
      paths:
        - /var/log/auth.log
        - /var/log/syslog
...
      document_type: syslog
...

Comment entire elasticsearch output section up to the Logstash as output.

filebeat.yml excerpt 2 of 2

...
  logstash:
    hosts: ["LOCAL_IP_LOGSTASH_SERVER:5044"]
...

# service filebeat restart

# update-rc.d filebeat defaults 95 10

Login to Kibana

change index parameter to filebeate-* and choose Time-field @timestamp
ELK_stack_2

PS:

To manage elasticserach you could install kopf plugin https://github.com/lmenezes/elasticsearch-kopf

on elsaticsearch server

# cd /usr/share/elasticsearch/

# ./bin/plugin install lmenezes/elasticsearch-kopf/2.x

open http://LOCAL_IP_ELASTIC_SERVER:9200/_plugin/kopf

to remove

# cd /usr/share/elasticsearch/

# ./bin/plugin remove kopf

login: http://LOCAL_IP_ELASTIC_SERVER:9200/_plugin/kopf/

2 thoughts on “Installing Logstash, Elasticsearch, Kibana (ELK stack) & Filebeat on Ubuntu 14.04”

  1. Hallo,

    thanks for the great tutorial.

    I found two mistakes

    The wget line with “wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –” does not work. Right is “wget -qO – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -“. I think the “-” is wrong.

    The secons mistake i found was in the echo command. Right is “echo deb https://packages.elastic.co/beats/apt stable main > /etc/apt/sources.list.d/beats.list”. The ” after echo and before > causes that the update does not work.

    Sry for my bad englisch, but i hope i can help someone.

  2. Hi, thank you for your good appraisal of my work, also thx for suggested improvements. I corrected the second mistake; could you explain more for the first mistake pls? What’s wrong with “-” (minus) ?

Leave a comment