opensearch - mata-elang-stable/mataelang-platform GitHub Wiki

Prerequisite
Configure Kernel
Setup OpenSearch
Dashboard Web UI
Useful Commands

opensearch

Example Environment

Item	Value
Kafka IP address	172.16.2.40
OpenSearch IP address	172.16.2.60
OpenSearch user	(user and password will be set below)

Prerequisite

✅ Ubuntu 20.04 LTS installed and updated with the following command.

sudo apt update && sudo apt -y upgrade

✅ Time Zone and NTP already set.

✅ Docker 20.10 or later installed with the following command.

sudo apt -y install docker.io

✅ Docker Compose 2.13 or later installed with the following command.

sudo curl -L "https://github.com/docker/compose/releases/download/v2.13.0/docker-compose-$(uname -s)-$(uname -m)"\
 -o /usr/bin/docker-compose && sudo chmod +x /usr/bin/docker-compose

Configure Kernel

▶️ Edit sysctl.conf to increase max virtual memory areas.

sudo nano /etc/sysctl.conf

Configuration

### Append to the end of the file.
vm.max_map_count=262144

▶️ Reload the kernel parameters.

sudo sysctl -p

✅ Check the max virtual memory areas.

cat /proc/sys/vm/max_map_count

Result

Setup OpenSearch

1. Configure OpenSearch

▶️ Clone Mata-Elang-Stable/opensearch-asset from GitHub to your server.

git clone https://github.com/mata-elang-stable/opensearch-asset.git ~/opensearch

▶️ Configure pipeline.conf.

sudo nano ~/opensearch/pipeline.conf

Configuration

🔑 Change input.kafka.bootstrap_servers to the Kafka server IP address and port number. (e.g. 172.16.2.40:9093)

🔑 Set output.opensearch.user and password for the initial Dashboard user.

input {
  kafka {
    bootstrap_servers => "172.17.0.1:9093"
  }
}

output {
  opensearch {
    user => "admin"
    password => "admin"
  }
}

2. Start OpenSearch

▶️ Start OpenSearch service.

sudo docker-compose -f ~/opensearch/docker-compose.yaml up -d

✅ Confirm the containers are running.

sudo docker-compose -f ~/opensearch/docker-compose.yaml ps -a

Result

NAME                    COMMAND                  SERVICE                 STATUS              PORTS
opensearch-dashboards   "./opensearch-dashbo…"   opensearch-dashboards   running             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp
opensearch-logstash     "/usr/local/bin/dock…"   opensearch-logstash     running             0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp, :::9600->9600/tcp, :::5000->5000/udp
opensearch-node1        "./opensearch-docker…"   opensearch-node1        running             9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp
opensearch-node2        "./opensearch-docker…"   opensearch-node2        running             9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp

Dashboard Web UI

1. Open Mata Elang Dashboard

▶️ Open the following URL to see the OpenSearch Dashboard.

URL: http://<OPENSEARCH_SERVER_IP_OR_NAME (e.g. 172.16.2.60)>:5601/

Click to view screen image

opensearch

2. Import Mata Elang Dashboard

▶️ Go to [Stack Management] -> [Saved Object] and import the file mata-elang-template.ndjson in Mata-Elang-Stable/opensearch-asset.

Click to view screen image

dashboard

Useful Commands

Click to show commands

Service Commands

✅ Show service status

sudo docker-compose -f ~/opensearch/docker-compose.yaml ps -a

Result

NAME                    COMMAND                  SERVICE                 STATUS              PORTS
opensearch-dashboards   "./opensearch-dashbo…"   opensearch-dashboards   running             0.0.0.0:5601->5601/tcp, :::5601->5601/tcp
opensearch-logstash     "/usr/local/bin/dock…"   opensearch-logstash     running             0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp, :::9600->9600/tcp, :::5000->5000/udp
opensearch-node1        "./opensearch-docker…"   opensearch-node1        running             9300/tcp, 9600/tcp, 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9650/tcp
opensearch-node2        "./opensearch-docker…"   opensearch-node2        running             9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp

✅ Start services

sudo docker-compose -f ~/opensearch/docker-compose.yaml up -d

✅ Stop services (and remove containers)

sudo docker-compose -f ~/opensearch/docker-compose.yaml down

✅ Stop services (and keep containers)

sudo docker-compose -f ~/opensearch/docker-compose.yaml stop

✅ Restart services

sudo docker-compose -f ~/opensearch/docker-compose.yaml restart

Maintenance Commands

✅ Show index list.

curl -u admin:admin -k -X GET https://localhost:9200/_cat/indices?v
curl -u admin:admin -k -X GET https://localhost:9200/_cat/indices/event-all-10s-*?v
curl -u admin:admin -k -X GET https://localhost:9200/_cat/indices/event-all-10s-2023.01.1*?v

✅ Show the number of objects in the index.

curl -u admin:admin -k -X GET https://localhost:9200/_cat/count/event-all-10s-*?v
curl -u admin:admin -k -X GET https://localhost:9200/_cat/count/event-all-10s-2023.01.20?v

✅ Query OpenSearch data.

curl -u admin:admin -k -X GET https://localhost:9200/event-all-10s-*/_search?pretty\
 -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must" : [{
        "range": {
          "@timestamp": {
            "gte": "2023-02-07T11:04:00+07:00",
            "lt": "2023-02-07T11:05:00+07:00"
          }
        }
      }, {
        "term": {
          "src_addr.keyword": "104.26.10.101"
        }
      }]
    }
  },
  "_source": [
    "@timestamp",
    "ip_id",
    "rule",
    "src_addr",
    "dst_addr"
  ]
}'

✅ Backup and Restore OpenSearch data.

Please prepare another host to backup date.
Full backup is NOT recommended. Specify the target date of backup and start it.

# update packages and install elasticdump
sudo apt update && sudo apt -y upgrade
sudo apt -y install nodejs npm
sudo npm install elasticdump -g

# Backup mapping and data of 2023.02.01
mkdir /home/ubuntu/backup
NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump\
 --input=https://admin:admin@<IP_OF_OPENSEARCH>:9200/event-all-10s-2023.02.01\
 --output=/home/ubuntu/backup/event-all-10s-2023.02.01-mapping.json --type=mapping
NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump\
 --input=https://admin:admin@<IP_OF_OPENSEARCH>:9200/event-all-10s-2023.02.01\
 --output=/home/ubuntu/backup/event-all-10s-2023.02.01.json --type=data

# Restore mapping and data
NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump\
 --input=/home/ubuntu/backup/event-all-10s-2023.02.01-mapping.json\
 --output=https://admin:admin@<IP_OF_OPENSEARCH>:9200/event-all-10s-2023.02.01 --type=mapping

NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump\
 --input=/home/ubuntu/backup/event-all-10s-2023.02.01.json\
 --output=https://admin:admin@<IP_OF_OPENSEARCH>:9200/event-all-10s-2023.02.01 --type=data

✅ Delete OpenSearch data.

# Delete data of January 2023
curl -u admin:admin -k -X DELETE https://localhost:9200/event-all-10s-2023.01.*

# Delete data by query that "rule" matches "116:281:1"
curl -u admin:admin -k -X POST https://localhost:9200/event-all-10s-*/_delete_by_query\
 -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "rule.keyword": "116:281:1"
    }
  }
}'

Configuration Commands

✅ Show environment variables

sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' opensearch-node1

sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' opensearch-dashboards

sudo docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' opensearch-logstash

✅ Show the loaded configurations

sudo docker-compose -f ~/opensearch/docker-compose.yaml exec opensearch-logstash\
 cat /usr/share/logstash/config/pipeline.conf

Log Commands

✅ Show OpenSearch log

sudo docker-compose -f ~/opensearch/docker-compose.yaml logs opensearch-node1

sudo docker-compose -f ~/opensearch/docker-compose.yaml logs opensearch-node2

sudo docker-compose -f ~/opensearch/docker-compose.yaml logs opensearch-dashboards

sudo docker-compose -f ~/opensearch/docker-compose.yaml logs opensearch-logstash

Version Commands

✅ Show OpenSearch version

curl -ku admin:admin https://localhost:9200/

✅ Show Logstash version

sudo docker-compose -f ~/opensearch/docker-compose.yaml exec opensearch-logstash logstash --version

✅ Show Docker version

sudo docker version

✅ Show Docker Compose version

docker-compose version

✅ Show OS version

cat /etc/os-release

Next Step >>

opensearch - mata-elang-stable/mataelang-platform GitHub Wiki

Contents

Example Environment

Prerequisite

Configure Kernel

Setup OpenSearch

1. Configure OpenSearch

2. Start OpenSearch

Dashboard Web UI

1. Open Mata Elang Dashboard

2. Import Mata Elang Dashboard

Useful Commands

Service Commands

Maintenance Commands

Configuration Commands

Log Commands

Version Commands

⚠️ GitHub.com Fallback ⚠️

opensearch - mata-elang-stable/mataelang-platform GitHub Wiki

Contents

Example Environment

Prerequisite

Configure Kernel

Setup OpenSearch

1. Configure OpenSearch

2. Start OpenSearch

Dashboard Web UI

1. Open Mata Elang Dashboard

2. Import Mata Elang Dashboard

Useful Commands

Service Commands

Maintenance Commands

Configuration Commands

Log Commands

Version Commands

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️