Set up Elasticsearch cluster on AWS on Debian for 10 steps

All commands and changes for faster implementation were done under root permissions. For the discovery of the nodes I use EC2 discovery plugin

 

1. Install Java

#echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee /etc/apt/sources.list.d/webupd8team-java.list

#echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list

#apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886

#apt-get update

#apt-get install oracle-java8-installer

#sed -i 's|JAVA_VERSION=8u151|JAVA_VERSION=8u162|' oracle-java8-installer.*

#sed -i 's|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/|PARTNER_URL=http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/|' oracle-java8-installer.*

#sed -i 's|SHA256SUM_TGZ="c78200ce409367b296ec39be4427f020e2c585470c4eed01021feada576f027f"|SHA256SUM_TGZ="68ec82d47fd9c2b8eb84225b6db398a72008285fafc98631b1ff8d2229680257"|' oracle-java8-installer.*

#sed -i 's|J_DIR=jdk1.8.0_151|J_DIR=jdk1.8.0_162|' oracle-java8-installer.*

#apt-get install oracle-java8-installer

#apt-get install oracle-java8-set-default

#java -version

#rm /etc/apt/sources.list.d/webupd8team-java.list

2. Install  Elasticsearch node

#wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch| sudo apt-key add -

#echo "deb https://artifacts.elastic.co/packages/5.x/aptstable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

#apt-get update && sudo apt-get install elasticsearch=5.6.3

installing EC2 discovery plugin

#/usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-ec2

3. Add drive to store ES data

format drive and add it to fstad

#cfdisk /dev/xvdb
#mkfs.ext4 /dev/xvdb1
#blkid /dev/xvdb1
#vim /etc/fstab
#UUID=   /opt ext4 rw,barrier=0,noatime,errors=remount-ro 0 2
#mount -a
#df -h

4. Prepare folders for data

#mkdir -p /opt/elasticsearch/data
#chown -R elasticsearch:elasticsearch /opt/elasticsearch/data
#chmod -R u+rw /opt/elasticsearch/data
#mkdir /var/log/elasticsearch
#chown -R elasticsearch:elasticsearch /var/log/elasticsearch

5. Configure ES properties

#vim /etc/elasticsearch/elasticsearch.yml
node.name: elastic-{node_number}
cluster.name: elastic-prod-1
network.host: _ec2_
discovery.zen.hosts_provider: ec2
cloud:
      aws:
           region: eu-central
           access_key: your_key
           secret_key: your_password
bootstrap.memory_lock: true
thread_pool.bulk.queue_size: 500
indices.memory.index_buffer_size: 20%
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /opt/elasticsearch/data
action.auto_create_index: false

6. Configure JVM properties for Elasticsearch

add for ES 30Gb of RAM

#vim /etc/elasticsearch/jvm.options
-Xms30g
-Xmx30g

7. Configure logging

insert such lines to log4j2.properties

#vim /etc/elasticsearch/log4j2.properties
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.condition.type = IfLastModified
appender.rolling.strategy.condition.age = 7D
appender.rolling.strategy.PathConditions.type = IfFileName
appender.rolling.strategy.PathConditions.glob = ${sys:es.logs.cluster_name}-*

change logging level from debug to info

logger.action.level = info

8. Configure service file

#mkdir -p /etc/systemd/system/elasticsearch.service.d/
#vim /etc/systemd/system/elasticsearch.service.d/override.conf
[Service]
LimitMEMLOCK=infinity

9. Change Swappines

#sysctl vm.swappiness=1

add a line to sysctl.conf

#vim /etc/sysctl.conf
vm.swappiness = 1

10. Reload and start service

#systemctl daemon-reload
#systemctl start elasticsearch.service

to check logs

#tail -f /var/log/elasticsearch/carerix-prod.log

Repeat for each server 😉

 

Useful stuff:

logs for failed service

#journalctl -u elasticsearch.service -b

files locations

#dpkg -L elasticsearch

Firefox extension Head for ES 

https://addons.mozilla.org/en-US/firefox/addon/elasticsearchhead/

Leave a comment