Set up Elasticsearch cluster on AWS on Debian for 10 steps

All commands and changes for faster implementation were done under root permissions. For the discovery of the nodes I use EC2 discovery plugin

 

1. Install Java

#echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list

#echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list

#apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886

#apt-get update

#apt-get install oracle-java8-installer

#sed -i 's|JAVA_VERSION=8u151|JAVA_VERSION=8u162|' oracle-java8-installer.*

#sed -i 's|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/|' oracle-java8-installer.*

#sed -i 's|SHA256SUM_TGZ="c78200ce409367b296ec39be4427f020e2c585470c4eed01021feada576f027f"|SHA256SUM_TGZ="68ec82d47fd9c2b8eb84225b6db398a72008285fafc98631b1ff8d2229680257"|' oracle-java8-installer.*

#sed -i 's|J_DIR=jdk1.8.0_151|J_DIR=jdk1.8.0_162|' oracle-java8-installer.*

#apt-get install oracle-java8-installer

#apt-get install oracle-java8-set-default

#java -version

#rm /etc/apt/sources.list.d/webupd8team-java.list

2. Install  Elasticsearch node

#wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch| sudo apt-key add -

#echo "deb https://artifacts.elastic.co/packages/5.x/aptstable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

#apt-get update && sudo apt-get install elasticsearch=5.6.3

installing EC2 discovery plugin

#/usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-ec2

3. Add drive to store ES data

format drive and add it to fstad

#cfdisk /dev/xvdb
#mkfs.ext4 /dev/xvdb1
#blkid /dev/xvdb1
#vim /etc/fstab
#UUID=   /opt ext4 rw,barrier=0,noatime,errors=remount-ro 0 2
#mount -a
#df -h

4. Prepare folders for data

#mkdir -p /opt/elasticsearch/data
#chown -R elasticsearch:elasticsearch /opt/elasticsearch/data
#chmod -R u+rw /opt/elasticsearch/data
#mkdir /var/log/elasticsearch
#chown -R elasticsearch:elasticsearch /var/log/elasticsearch

5. Configure ES properties

#vim /etc/elasticsearch/elasticsearch.yml
node.name: elastic-{node_number}
cluster.name: elastic-prod-1
network.host: _ec2_
discovery.zen.hosts_provider: ec2
cloud:
      aws:
           region: eu-central
           access_key: your_key
           secret_key: your_password
bootstrap.memory_lock: true
thread_pool.bulk.queue_size: 500
indices.memory.index_buffer_size: 20%
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /opt/elasticsearch/data
action.auto_create_index: false

6. Configure JVM properties for Elasticsearch

add for ES 30Gb of RAM

#vim /etc/elasticsearch/jvm.options
-Xms30g
-Xmx30g

7. Configure logging

insert such lines to log4j2.properties

#vim /etc/elasticsearch/log4j2.properties
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.condition.type = IfLastModified
appender.rolling.strategy.condition.age = 7D
appender.rolling.strategy.PathConditions.type = IfFileName
appender.rolling.strategy.PathConditions.glob = ${sys:es.logs.cluster_name}-*

change logging level from debug to info

logger.action.level = info

8. Configure service file

#mkdir -p /etc/systemd/system/elasticsearch.service.d/
#vim /etc/systemd/system/elasticsearch.service.d/override.conf
[Service]
LimitMEMLOCK=infinity

9. Change Swappines

#sysctl vm.swappiness=1

add a line to sysctl.conf

#vim /etc/sysctl.conf
vm.swappiness = 1

10. Reload and start service

#systemctl daemon-reload
#systemctl start elasticsearch.service

to check logs

#tail -f /var/log/elasticsearch/carerix-prod.log

Repeat for each server 😉

 

Useful stuff:

logs for failed service

#journalctl -u elasticsearch.service -b

files locations

#dpkg -L elasticsearch

Firefox extension Head for ES 

https://addons.mozilla.org/en-US/firefox/addon/elasticsearchhead/

Get UUID of Hard Disks on Debian

1st option
# ls -l /dev/disk/by-uuid/

lrwxrwxrwx 1 root root 15 Nov 7 13:00 9c18302c-9aa2-4f55-b2c8-03a9ae68b2a2 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Nov 7 13:00 b524f8c0-90e7-4fc7-a842-6cb2380086c8 -> ../../nvme0n1p1

2nd option

# blkid /dev/nvme1n1p1

/dev/nvme1n1p1: UUID="9c18302c-9aa2-4f55-b2c8-03a9ae68b2a2" TYPE="ext4" PARTUUID="1e534c5e-e4c8-424c-822b-117e47d72263"

Adding routes to Debian

This adds the route immediately to the Kernel IP routing table (after reboot it will be erased).

# route add -net 172.31.0.0 netmask 255.255.0.0 gw 192.168.1.1

to print the Kernel IP Routing table:

# netstat -rn

To keep the Static Route persistent you need to edit the file: /etc/network/interfaces

and the static routes in the following format:

up route add [-net/-host] <host/net>/<mask> gw <host/ip> dev <interface>

up route add -net 172.31.0.0/16 gw 192.168.1.1 dev eno1

 

Setting the exact time Debian 6

# apt-get install ntpdate
# dpkg-reconfigure tzdata

Synchronise with time server:
# ntpdate pool.ntp.org

If all done good, then you will see the correct current time corresponding to your time zone in the screen.

To live with out manual correction of the time, the last command can be put in the /etc/crontab to update rate, for example once a week.

42 4    * * 0   root    /usr/sbin/ntpdate pool.ntp.org

Will run every Sunday at 4:42 am