Real-Time Threat Detection with Elastic Stack PART-1

Practical Insights into Rapid Threat Detection with Elastic Stack, DataDog, and Graylog (Setting up of hunting lab)

This blog series will cover:

• Collecting Data

Data Collection

• Visualizing and Exploring Data

Data Visualisation

• Automating Threat Detection

Threat detection

• Monitoring Security and Threats

Datadog

• Illustrating the Threat Landscape

graylog

Decoding Threat Intelligence

The process of identifying, analyzing, and understanding threats that have targeted the organization or similar organizations in the past. Indicators of Compromise(I0Cs) are frequently shared online for anybody to use and potentially uncover similar patterns in their own security.

By using Threat Intelligence, we seek to understand the methods used by attackers, vulnerabilities found in the network, system or applications.
We also seek to identify who our adversaries are and where they’re attacking us from.

Part 1: ELK Stack Installation on Server or Virtual Machine

Step-by-Step Guide: Detailed instructions on installing Elasticsearch, Logstash, and Kibana on various server environments and virtual machines.

Configuration Tips: Best practices for configuring and optimizing the ELK Stack for efficient performance and scalability.

Part 2: Data Collection with Beats, DataDog, and Graylog

Beats Integration: How to set up and configure Beats to collect and ship data from various sources to the ELK Stack.

DataDog and Graylog Integration: Methods to integrate DataDog and Graylog for enhanced data monitoring, analysis, and visualization.

Setting Up Your Hunting Lab

Now, let’s dive into the steps for Part 1.

Step 1 : Installing Dependencies

sudo apt update
sudo apt install default-jdk
sudo apt install default-jre

Step 2 : Adding Elastic Repository

1. Import the PGP key for Elastic :

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

2. Install the apt-trasport-https package :

sudo apt-get install apt-transport-https

3. Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list:

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

4. Update the repo :

sudo apt update

Step 3 : Install Elasticsearch

1. Install Elasticsearch:

sudo apt-get update && sudo apt-get install elasticsearch

elasticsearch installation

2. Configure Elasticsearch:

sudo nano /etc/elasticsearch/elasticsearch.yml

• Scroll down and find

#network.host: 192.168.0.1
#http.port: 9200

• Uncomment those lines by deleting the hashes (’#’).
• Replace 192.168.0.1 with 0.0.0.0

Just below, find the Discovery section,
We are adding one more line .

discovery.seed_hosts: []

elasticsearch.yml configuration

Change the default true to xpack.security.enabled:false

3. Start Elasticsearch:

sudo systemctl start elasticsearch.service

To start Elasticsearch on boot :

sudo systemctl enable elasticsearch.service

Active(running)

4. Test Elasticsearch

• Go to http://<server-ip>:9200/

localhost:9200

Install Elasticsearch with Debian Package | Elasticsearch Guide [8.14] | Elastic

Step 4 : Install Kibana

1. Install Kibana

sudo apt-get install kibana

2. Configure Kibana

Open kibana.yml config file :

sudo nano /etc/kibana/kibana.yml

Uncomment by deleting hashes on the following lines :

#server.port: 5601
#server.name: “your-hostname”
#elasticsearch.hosts: [“http://<server-ip>:9200"]

kibana.yml

3. Start Kibana

sudo systemctl start kibana

To launch at boot :

sudo systemctl enable kibana

4. Test Kibana

Go to http://localhost:5601

Step 5 : Install Logstash

1. Install Logstash

sudo apt install logstash

1. Start Logstash

sudo systemctl start logstash
sudo systemctl enable logstash

Step 6 : Install Filebeat

💡 Filebeat is a lightweight plugin designed to collect and ship log files. As the most commonly used Beats module, Filebeat offers several advantages:

Efficiency: It efficiently collects and forwards log data to Logstash or Elasticsearch, ensuring minimal resource usage.

Backpressure Handling: One of its key features is the ability to adjust its data transmission rate if the Logstash service becomes overwhelmed, preventing data loss and ensuring smooth operations.

Reliability: Filebeat guarantees data delivery with its built-in mechanisms for handling network issues and ensuring logs are not lost during transmission.

sudo apt-get install filebeat

install filebeat

2. Configure Filebeat

sudo nano /etc/filebeat/filebeat.yml

• Comment the following lines :

output.elasticsearch:
    Array of hosts to connect to.
    hosts: ["localhost:9200"]

• Uncomment these following lines :

# output.logstash
     # hosts: ["localhost:5044"]

filebeat.yml

Enable the Filebeat system module, which will examine local system logs :

sudo filebeat modules enable system

enable filebeat

1. Load Index Template :

sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

2. Start Filebeat

sudo systemctl start filebeat

3. Verify Elasticsearch Reception of Data

curl -XGET http://localhost:9200/_cat/indices?v

test connection

Stay tuned for the upcoming second part, where we delve deeper into advanced techniques and strategies for real-time threat detection and mitigation where we will be using victim machine to test a realtime threat detection .

References:

Filebeat service failing to start

What is Elasticsearch? | Elasticsearch Guide [8.14] | Elastic