Elasticsearch Installation (v7.12.0)

1. Download and Extract

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz
tar xvf elasticsearch-7.12.0-linux-x86_64.tar.gz
mv elasticsearch-7.12.0 elasticsearch

2. Start Elasticsearch

➡️ Go to the Elasticsearch directory:

cd elasticsearch

✅ Recommended: run as a daemon:

bin/elasticsearch -d -p pid

⚠️ If it fails, try running in foreground to view logs:

bin/elasticsearch

3. Run as a Systemd Service

Create service file:

sudo nano /usr/lib/systemd/system/elastic.service

Paste:

[Unit]
Description=Elastic

[Service]
RuntimeDirectory=/home/hadoop/elasticsearch
Environment=ES_HOME=/home/hadoop/elasticsearch
Environment=ES_PATH_CONF=/home/hadoop/elasticsearch/config
Environment=ES_HEAP_SIZE=512M
Environment=ES_JAVA_OPTS=-Xmx2g -Xms2g
Type=simple
User=hadoop
Group=hadoop
ExecStart=/home/hadoop/elasticsearch/bin/elasticsearch
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable elastic.service
sudo systemctl start elastic.service
sudo systemctl status elastic.service

4. Test Connection

curl -X GET "http://localhost:9200/?pretty"

5. Troubleshooting

If the service fails to start, check logs:

sudo journalctl -u elastic.service -b

---

Kibana Installation (v7.12.0)

1. Download and Extract Kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.12.0-linux-x86_64.tar.gz
tar xvf kibana-7.12.0-linux-x86_64.tar.gz
mv kibana-7.12.0-linux-x86_64 kibana
cd kibana

2. Configure Kibana

Edit the config:

nano config/kibana.yml

Uncomment and update the following lines:

• Line 32:

elasticsearch.hosts: ["http://localhost:9200"]

• Line 50:

server.ssl.enabled: false

---

3. Run Kibana as a Service

Create the systemd service:

sudo nano /usr/lib/systemd/system/kibana.service

Paste:

[Unit]
Description=Kibana

[Service]
Type=simple
User=hadoop
Group=hadoop
ExecStart=/home/hadoop/kibana/bin/kibana -c /home/hadoop/kibana/config/kibana.yml
Restart=always

[Install]
WantedBy=multi-user.target

Reload and start the service:

sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service
sudo systemctl status kibana.service

4. Accessing Kibana (after Elasticsearch is running)

Allow port through firewall:

sudo ufw allow 5601/tcp

Tunnel port 5601 from your machine to the VM:

ssh -N -L 5601:localhost:5601 [email protected] -vv

Then open in your browser:

http://localhost:5601

You'll see the Kibana dashboard if everything is configured correctly.

Logstash Installation (v7.12.0)

1. Download and Extract

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.12.0-linux-x86_64.tar.gz
tar xvf logstash-7.12.0-linux-x86_64.tar.gz
mv logstash-7.12.0 logstash
cd logstash

2. Basic Test Pipeline

Test Logstash with a basic stdin → stdout pipeline:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

Type a message (e.g. hello world) and you should get a structured JSON response.

Exit with Ctrl + D.

3. Optional: Run a Config File

Create a file first-pipeline.conf:

nano first-pipeline.conf

Example:

input {
  stdin { }
}
output {
  stdout { codec => rubydebug }
}

Run it:

bin/logstash -f first-pipeline.conf --config.test_and_exit
bin/logstash -f first-pipeline.conf --config.reload.automatic

Filebeat Installation (v7.12.0)

1. Download and Extract Filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-linux-x86_64.tar.gz
tar xvf filebeat-7.12.0-linux-x86_64.tar.gz
mv filebeat-7.12.0-linux-x86_64 filebeat
cd filebeat

2. Configure Filebeat

Edit config:

nano filebeat.yml

Set these minimal values:

output.elasticsearch:
  hosts: ["localhost:9200"]

setup.kibana:
  host: "localhost:5601"

Uncomment and adjust the input path if needed:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/hadoop/logs/*.log

3. Run Filebeat

To run in the foreground:

./filebeat -e

To test configuration:

./filebeat test config

To run as a service (manual approach):

nohup ./filebeat > filebeat.log 2>&1 &

4. Optional: Load Dashboards (Requires Kibana)

./filebeat setup --dashboards

./filebeat setup --index-management
./filebeat setup --templates