Real-Time Threat Detection with Elastic Stack PART-1
Practical Insights into Rapid Threat Detection with Elastic Stack, DataDog, and Graylog (Setting up of hunting lab)
This blog series will cover:
- Collecting Data
- Visualizing and Exploring Data
- Automating Threat Detection
- Monitoring Security and Threats
- Illustrating the Threat Landscape
Decoding Threat Intelligence
The process of identifying, analyzing, and understanding threats that have targeted the organization or similar organizations in the past. Indicators of Compromise(I0Cs) are frequently shared online for anybody to use and potentially uncover similar patterns in their own security.
By using Threat Intelligence, we seek to understand the methods used by attackers, vulnerabilities found in the network, system or applications.
We also seek to identify who our adversaries are and where they’re attacking us from.
Part 1: ELK Stack Installation on Server or Virtual Machine
Step-by-Step Guide: Detailed instructions on installing Elasticsearch, Logstash, and Kibana on various server environments and virtual machines.
Configuration Tips: Best practices for configuring and optimizing the ELK Stack for efficient performance and scalability.
Part 2: Data Collection with Beats, DataDog, and Graylog
Beats Integration: How to set up and configure Beats to collect and ship data from various sources to the ELK Stack.
DataDog and Graylog Integration: Methods to integrate DataDog and Graylog for enhanced data monitoring, analysis, and visualization.
Setting Up Your Hunting Lab
Now, let’s dive into the steps for Part 1.
Step 1 : Installing Dependencies
sudo apt update
sudo apt install default-jdk
sudo apt install default-jre
Step 2 : Adding Elastic Repository
- Import the PGP key for Elastic :
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
2. Install the apt-trasport-https package :
sudo apt-get install apt-transport-https
3. Save the repository definition to /etc/apt/sources.list.d/elastic-8.x.list
:
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
4. Update the repo :
sudo apt update
Step 3 : Install Elasticsearch
1. Install Elasticsearch:
sudo apt-get update && sudo apt-get install elasticsearch
2. Configure Elasticsearch:
sudo nano /etc/elasticsearch/elasticsearch.yml
- Scroll down and find
#network.host: 192.168.0.1
#http.port: 9200
- Uncomment those lines by deleting the hashes (’#’).
- Replace 192.168.0.1 with 0.0.0.0
Just below, find the Discovery section,
We are adding one more line .
discovery.seed_hosts: []
Change the default true to xpack.security.enabled:false
3. Start Elasticsearch:
sudo systemctl start elasticsearch.service
To start Elasticsearch on boot :
sudo systemctl enable elasticsearch.service
4. Test Elasticsearch
- Go to http://<server-ip>:9200/
Step 4 : Install Kibana
1. Install Kibana
sudo apt-get install kibana
2. Configure Kibana
Open kibana.yml config file :
sudo nano /etc/kibana/kibana.yml
Uncomment by deleting hashes on the following lines :
#server.port: 5601
#server.name: “your-hostname”
#elasticsearch.hosts: [“http://<server-ip>:9200"]
3. Start Kibana
sudo systemctl start kibana
To launch at boot :
sudo systemctl enable kibana
4. Test Kibana
Go to http://localhost:5601
Step 5 : Install Logstash
- Install Logstash
sudo apt install logstash
1. Start Logstash
sudo systemctl start logstash
sudo systemctl enable logstash
Step 6 : Install Filebeat
💡 Filebeat is a lightweight plugin designed to collect and ship log files. As the most commonly used Beats module, Filebeat offers several advantages:
Efficiency: It efficiently collects and forwards log data to Logstash or Elasticsearch, ensuring minimal resource usage.
Backpressure Handling: One of its key features is the ability to adjust its data transmission rate if the Logstash service becomes overwhelmed, preventing data loss and ensuring smooth operations.
Reliability: Filebeat guarantees data delivery with its built-in mechanisms for handling network issues and ensuring logs are not lost during transmission.
sudo apt-get install filebeat
2. Configure Filebeat
sudo nano /etc/filebeat/filebeat.yml
- Comment the following lines :
output.elasticsearch:
Array of hosts to connect to.
hosts: ["localhost:9200"]
- Uncomment these following lines :
# output.logstash
# hosts: ["localhost:5044"]
Enable the Filebeat system module, which will examine local system logs :
sudo filebeat modules enable system
- Load Index Template :
sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
2. Start Filebeat
sudo systemctl start filebeat
3. Verify Elasticsearch Reception of Data
curl -XGET http://localhost:9200/_cat/indices?v
Stay tuned for the upcoming second part, where we delve deeper into advanced techniques and strategies for real-time threat detection and mitigation where we will be using victim machine to test a realtime threat detection .