Elastic Cluster - cchantra/bigdata.github.io GitHub Wiki
https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html
When you start an instance of Elasticsearch, you are starting a node. An Elasticsearch cluster is a group of nodes that have the same cluster.name attribute. As nodes join or leave a cluster, the cluster automatically reorganizes itself to evenly distribute the data across the available nodes.
If you are running a single instance of Elasticsearch, you have a cluster of one node. All primary shards reside on the single node. No replica shards can be allocated, therefore the cluster state remains yellow. The cluster is fully functional but is at risk of data loss in the event of a failure.
As the name implies, an Elasticsearch cluster is a group of one or more Elasticsearch nodes instances that are connected together. The power of an Elasticsearch cluster lies in the distribution of tasks, searching and indexing, across all the nodes in the cluster.
The nodes in the Elasticsearch cluster can be assigned different jobs or responsibilities:
-
Data nodes — store data and execute data-related operations such as search and aggregation
-
Master nodes — in charge of cluster-wide management and configuration actions such as adding and removing nodes
-
Client nodes — forwards cluster requests to the master node and data-related requests to data nodes
-
Ingest nodes — for pre-processing documents before indexing
Note: Tribe nodes, which were similar to cross-cluster or federated nodes, were deprecated with since Elasticsearch 5.4
By default, each node is automatically assigned a unique identifier, or name, that is used for management purposes and becomes even more important in a multi-node, or clustered, environment.
Proceed as normal installation; Do it for all nodes.
NOTE *** : use version 8.10 for the following one.
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.10.0-linux-x86_64.tar.gz
-untar
tar xvf ....
mv elasticsearch-8.10.0 elasticsearch
-rename folders
cd elasticsearch
And configure as follows:
set up the cluster so that the nodes can connect and communicate with each other.
For each node, open the Elasticsearch configuration file: sudo vim /home/hadoop/elasticsearch/elasticsearch.yml
This file is quite long, and contains multiple settings for different sections. Browse through the file, and enter the following configurations (replace the IPs with your node IPs): (you can explore the file)
#give your cluster a name.
Cluster.name: es-cluster
#give your nodes a name (change node number from node to node).
node.name: "server1" # this the server name for each node in /etc/hosts
#enter the private IP and port of your node: (all nodes)
network.host: 0.0.0.0
http.port: 9200
#detail your nodes for discovery:
discovery.seed_hosts: ["server1", "server2"]
cluster.initial_master_nodes: ["server1"]
#by pass security
xpack.security.enabled: false
Save and exit.
using vim
vi conf/jvm.options
Add the following to correct the error on VM setting
-Xms2g
-Xms2g
-Xmx2g
Update VM memory map area
sudo sysctl -w vm.max_map_count=262144
You are now ready to start your Elasticsearch nodes and verify they are communicating with each other as a cluster. For each instance, run the following command:
sudo systemctl start elastic.service
If everything was configured correctly, your Elasticsearch cluster should be up and running. To verify everything is working as expected, query Elasticsearch from any of the cluster nodes:
curl -XGET 'http://<serverip>:9200/_cluster/state?pretty'
The response should detail the cluster and its nodes:
{
"cluster_name" : "elastic-cluster",
"cluster_uuid" : "CvtoaPwARqmRFvK4FHFFww",
"version" : 35,
"state_uuid" : "U_3SuKpbTnWpfULWSgCoTA",
"master_node" : "CD75d_ytREm30_Wy2V1spQ",
"blocks" : { },
"nodes" : {
"CD75d_ytREm30_Wy2V1spQ" : {
"name" : "server1",
"ephemeral_id" : "k407AgurSmqDf_D9zhT6Qg",
"transport_address" : "10.3.135.170:9300",
"external_id" : "server1",
"attributes" : {
"ml.allocated_processors_double" : "4.0",
"ml.machine_memory" : "16765722624",
"ml.config_version" : "10.0.0",
….
"GAC92IYNTeaile7i-d_ueg" : {
"name" : "server2",
"ephemeral_id" : "OY1mHPtYSqW2JkcYx0DWRg",
"transport_address" : "10.3.135.169:9300",
"external_id" : "server2",
"attributes" : {
"ml.allocated_processors_double" : "4.0",
"ml.machine_memory" : "12483362816",
"ml.config_version" : "10.0.0",
…
Or type to check the number of nodes
curl -XGET 'localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "elastic-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
eg. cannot connect to server:9200 connection refused. Look at logs at logs/elastic-cluster.log
For example,
tail -500 logs/elastic-cluster.log
Inspect the error the error message Elasticsearch cluster configurations for production We already defined the different roles for the nodes in our cluster, but there are some additional recommended settings for a cluster running in a production environment.
Avoiding "Split Brain"
A “split-brain” situation is when communication between nodes in the cluster fails due to either a network failure or an internal failure with one of the nodes. In this kind of scenario, more than one node might believe it is the master node, leading to a state of data inconsistency.
For avoiding this situation, we can make changes to the discovery.zen.minimum_master_nodes directive in the Elasticsearch configuration file which determines how many nodes need to be in communication (quorum) to elect a master. A best practice to determine this number is to use the following formula to decide this number: N/2 + 1. N is the number of master eligible nodes in the cluster. You then round down the result to the nearest integer.
In the case of a cluster with three nodes, then:
discovery.zen.minimum_master_nodes: 2
Adjusting JVM heap size
To ensure Elasticsearch has enough operational leeway, the default JVM heap size (min/max 1 GB) should be adjusted. As a rule of the thumb, the maximum heap size should be set up to 50% of your RAM, but no more than 32GB (due to Java pointer inefficiency in larger heaps). Elastic also recommends that the value for maximum and minimum heap size be identical.
These value can be configured using the Xmx and Xms settings in the jvm.options file. On DEB:
sudo vim /etc/elasticsearch/jvm.options
add
-Xms2g
-Xms2g
-Xmx2g
sysctl -w vm.max_map_count=262144
Disabling swapping
Swapping out unused memory is a known behavior but in the context of Elasticsearch can result in disconnects, bad performance and in general — an unstable cluster.
To avoid swapping you can either disable all swapping (recommended if Elasticsearch is the only service running on the server), or you can use mlockall to lock the Elasticsearch process to RAM. To do this, open the Elasticsearch configuration file on all nodes in the cluster:
sudo vim /etc/elasticsearch/elasticsearch.yml
Uncomment the following line:
bootstrap.mlockall: true
Next, open the /etc/default/elasticsearch file:
sudo vim /etc/default/elasticsearch
Make the following configurations:
MAX_LOCKED_MEMORY=unlimited
Restart Elasticsearch when you’re done.
Adjusting virtual memory.
To avoid running out of virtual memory, increase the amount of limits on mmap counts:
sudo vim /etc/sysctl.conf
Update the relevant setting accordingly:
vm.max_map_count=262144
On DEB/RPM, this setting is configured automatically. Increasing open file descriptor limit Another important configuration is the limit of open file descriptors. Since Elasticsearch makes use of a large amount of file descriptors, you must ensure the defined limit is enough otherwise you might end up losing data. The common recommendation for this setting is 65,536 and higher. On DEB/RPM the default settings are already configured to suit this requirement but you can of course fine tune it.
sudo vim /etc/security/limits.conf
Set the limit:
- nofile 65536
Elasticsearch supports a large number of cluster-specific API operations that allow you to manage and monitor your Elasticsearch cluster. Most of the APIs allow you to define which Elasticsearch node to call using either the internal node ID, its name or its address.
Below is a list of a few of the more basic API operations you can use. For advanced usage of cluster APIs, read this blog post. (https://logz.io/blog/elasticsearch-cheat-sheet/)
This API can be used to see general info on the cluster and gauge its health:
curl -XGET 'localhost:9200/_cluster/health?pretty'
Response:
{
"cluster_name" : "my-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
This API can be sued to see a detailed status report on your entire cluster. You can filter results by specifying parameters in the call URL.
curl -XGET 'localhost:9200/_cluster/state?pretty'
Response:
{
"cluster_name" : "my-cluster",
"compressed_size_in_bytes" : 347,
"version" : 4,
"state_uuid" : "uMi5OBtAS8SSRJ9hw1-gUg",
"master_node" : "sqT_y5ENQ9SdjHiE0oco_g",
"blocks" : { },
"nodes" : {
"sqT_y5ENQ9SdjHiE0oco_g" : {
"name" : "node-1",
"ephemeral_id" : "-HDzovR0S0e-Nn8XJ-GWPA",
"transport_address" : "172.31.56.131:9300",
"attributes" : { }
},
"mO0d0hYiS1uB--NoWuWyHg" : {
"name" : "node-3",
"ephemeral_id" : "LXjx86Q5TrmefDoq06MY1A",
"transport_address" : "172.31.58.61:9300",
"attributes" : { }
},
….
Extremely useful for monitoring performance metrics on your entire cluster:
curl -XGET 'localhost:9200/_cluster/stats?human&pretty'
Response:
{
"_nodes" : {
"total" : 3,
"successful" : 3,
"failed" : 0
},
"cluster_name" : "my-cluster",
"timestamp" : 1517224098451,
"status" : "green",
"indices" : {
"count" : 0,
"shards" : { },
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size" : "0b",
"size_in_bytes" : 0
},
"fielddata" : {
"memory_size" : "0b",
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "0b",
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 0,
"memory" : "0b",
"memory_in_bytes" : 0,
"terms_memory" : "0b",
"terms_memory_in_bytes" : 0,
"stored_fields_memory"
….
If you want to inspect metrics for specific nodes in the cluster, use this API. You can see info for all nodes, a specific node, or ask to see only index or OS/process specific stats. All nodes:
curl -XGET 'localhost:9200/_nodes/stats?pretty'
A specific node:
curl -XGET 'localhost:9200/_nodes/node-1/stats?pretty'
Index-only stats:
curl -XGET 'localhost:9200/_nodes/stats/indices?pretty'
You can get any of the specific metrics for any single node with the following structure:
curl -XGET 'localhost:9200/_nodes/stats/ingest?pretty'
Or multiple nodes with the following structure:
curl -XGET 'localhost:9200/_nodes/stats/ingest,fs?pretty'
Or all metrics with either of these two formats:
curl -XGET 'localhost:9200/_nodes/stats/_all?pretty'
curl -XGET 'localhost:9200/_nodes/stats?metric=_all?pretty'
More:
https://logz.io/blog/elasticsearch-cluster-tutorial/
Download kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.10.1-linux-x86_64.tar.gz
Untar it Change folder name
mv kibana-8.10.1 kibana
cd kibana
Open the kibana.yml file: (please explore the config file)
vim config/ kibana.yml
uncomment the following line:
elasticsearch.hosts: ["http://server1:9200/"] # you may add more elastic master here
Change the line
#server.host: "localhost"
To
server.host: 0.0.0.0 # loop back to allow remote access
Also config disable openssl in node.options
vi config/node.options and comment the option
#--openssl-legacy-provider
Save it.
Run as a service
====create service file where hadoop is user name and group name
sudo vi /usr/lib/systemd/system/kibana.service
[Unit]
Description=Kibana
[Service]
Type=simple
User=hadoop
Group=hadoop
#User=root
#Group=root
# location of your kibana
ExecStart=/home/hadoop/kibana/bin/kibana serve -c /home/hadoop/kibana/config/kibana.yml Restart=always
[Install]
WantedBy=multi-user.target
=== save it
Reload service file, enable at start and start service, check status
sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service
sudo systemctl status kibana.service
Check your status , wait until it says “available”
sudo journalctl -n 100 -u kibana.service
Allow port
sudo ufw allow 5601/tcp
Then tunnel to localhost port 5601
ssh -N -L 5601:localhost:5601 [email protected]
Navigate to : see the sample dashboard
Try elasticsearch using UI
http://localhost:5601/app/kibana#/dev_tools/console?_g=()
1 Register ngrok account
https://dashboard.ngrok.com/login
- Login to your account
Download "linux" at 1) in the following figure.
3 unpack at 2)
- Go to get your token at 3) in the page below.
-
Configure your ngrok token in your vm at 4)
-
Run your ngrok on your vm
ngrok http 5601
- Run
curl -s http://localhost:4040/api/tunnels | python3 -c "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
And copy your web address
Copyright (c) by Chantana chantrapornchai for big data platform class.