Set up Elasticsearch cluster on AWS on Debian for 10 steps

All commands and changes for faster implementation were done under root permissions. For the discovery of the nodes I use EC2 discovery plugin

 

1. Install Java

#echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list

#echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list

#apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886

#apt-get update

#apt-get install oracle-java8-installer

#sed -i 's|JAVA_VERSION=8u151|JAVA_VERSION=8u162|' oracle-java8-installer.*

#sed -i 's|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/|' oracle-java8-installer.*

#sed -i 's|SHA256SUM_TGZ="c78200ce409367b296ec39be4427f020e2c585470c4eed01021feada576f027f"|SHA256SUM_TGZ="68ec82d47fd9c2b8eb84225b6db398a72008285fafc98631b1ff8d2229680257"|' oracle-java8-installer.*

#sed -i 's|J_DIR=jdk1.8.0_151|J_DIR=jdk1.8.0_162|' oracle-java8-installer.*

#apt-get install oracle-java8-installer

#apt-get install oracle-java8-set-default

#java -version

#rm /etc/apt/sources.list.d/webupd8team-java.list

2. Install  Elasticsearch node

#wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch| sudo apt-key add -

#echo "deb https://artifacts.elastic.co/packages/5.x/aptstable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

#apt-get update && sudo apt-get install elasticsearch=5.6.3

installing EC2 discovery plugin

#/usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-ec2

3. Add drive to store ES data

format drive and add it to fstad

#cfdisk /dev/xvdb
#mkfs.ext4 /dev/xvdb1
#blkid /dev/xvdb1
#vim /etc/fstab
#UUID=   /opt ext4 rw,barrier=0,noatime,errors=remount-ro 0 2
#mount -a
#df -h

4. Prepare folders for data

#mkdir -p /opt/elasticsearch/data
#chown -R elasticsearch:elasticsearch /opt/elasticsearch/data
#chmod -R u+rw /opt/elasticsearch/data
#mkdir /var/log/elasticsearch
#chown -R elasticsearch:elasticsearch /var/log/elasticsearch

5. Configure ES properties

#vim /etc/elasticsearch/elasticsearch.yml
node.name: elastic-{node_number}
cluster.name: elastic-prod-1
network.host: _ec2_
discovery.zen.hosts_provider: ec2
cloud:
      aws:
           region: eu-central
           access_key: your_key
           secret_key: your_password
bootstrap.memory_lock: true
thread_pool.bulk.queue_size: 500
indices.memory.index_buffer_size: 20%
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /opt/elasticsearch/data
action.auto_create_index: false

6. Configure JVM properties for Elasticsearch

add for ES 30Gb of RAM

#vim /etc/elasticsearch/jvm.options
-Xms30g
-Xmx30g

7. Configure logging

insert such lines to log4j2.properties

#vim /etc/elasticsearch/log4j2.properties
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.condition.type = IfLastModified
appender.rolling.strategy.condition.age = 7D
appender.rolling.strategy.PathConditions.type = IfFileName
appender.rolling.strategy.PathConditions.glob = ${sys:es.logs.cluster_name}-*

change logging level from debug to info

logger.action.level = info

8. Configure service file

#mkdir -p /etc/systemd/system/elasticsearch.service.d/
#vim /etc/systemd/system/elasticsearch.service.d/override.conf
[Service]
LimitMEMLOCK=infinity

9. Change Swappines

#sysctl vm.swappiness=1

add a line to sysctl.conf

#vim /etc/sysctl.conf
vm.swappiness = 1

10. Reload and start service

#systemctl daemon-reload
#systemctl start elasticsearch.service

to check logs

#tail -f /var/log/elasticsearch/carerix-prod.log

Repeat for each server 😉

 

Useful stuff:

logs for failed service

#journalctl -u elasticsearch.service -b

files locations

#dpkg -L elasticsearch

Firefox extension Head for ES 

https://addons.mozilla.org/en-US/firefox/addon/elasticsearchhead/

Linux exit codes

To get the exit code of a command type echo $? at the command prompt. In the following example a file is printed to the terminal using the cat command.

cat file.txt 
hello world 
echo $? 
0

The command was successful. The file exists and there are no errors in reading the file or writing it to the terminal. The exit code is therefor 0.

cat doesnotexist.txt 
cat: doesnotexist.txt: No such file or directory 
echo $? 
1

The exit code is 1 as the operation was not successful.

To use exit codes in scripts an “if” statement can be used to see if an operation was successful.

#!/bin/bash 
cat file.txt 

if [ $? -eq 0 ] 
then 
 echo "The script ran ok" 
 exit 0 
else 
 echo "The script failed" >&2 
 exit 1 
fi

To set an exit code in a script use “exit0” where “0” is the number you want to return. In the following example a shell script exits with a “1”. This file is saved as “exit.sh”.

#!/bin/bash 
exit 1
bash exit.sh 
echo $? 
1

Get UUID of Hard Disks on Debian

1st option
# ls -l /dev/disk/by-uuid/

lrwxrwxrwx 1 root root 15 Nov 7 13:00 9c18302c-9aa2-4f55-b2c8-03a9ae68b2a2 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Nov 7 13:00 b524f8c0-90e7-4fc7-a842-6cb2380086c8 -> ../../nvme0n1p1

2nd option

# blkid /dev/nvme1n1p1

/dev/nvme1n1p1: UUID="9c18302c-9aa2-4f55-b2c8-03a9ae68b2a2" TYPE="ext4" PARTUUID="1e534c5e-e4c8-424c-822b-117e47d72263"

Adding routes to Debian

This adds the route immediately to the Kernel IP routing table (after reboot it will be erased).

# route add -net 172.31.0.0 netmask 255.255.0.0 gw 192.168.1.1

to print the Kernel IP Routing table:

# netstat -rn

To keep the Static Route persistent you need to edit the file: /etc/network/interfaces

and the static routes in the following format:

up route add [-net/-host] <host/net>/<mask> gw <host/ip> dev <interface>

up route add -net 172.31.0.0/16 gw 192.168.1.1 dev eno1