Huntress Rebuilds Its Detection Engine to Process Endpoint Data in Real Time
Summary
Huntress describes replacing scheduled Elasticsearch detections with a custom streaming engine and hosted OpenSearch, aiming to reduce detection delays, remove a 100-result limit, and lower infrastructure costs.
Key points
- Huntress says its pipeline handles data from more than two million endpoints, at an estimated 30 TB per day.
- Its previous setup sent endpoint survey data through processing and Elasticsearch, where scheduled Sigma rules generated signals for SOC investigations.
- Detection could take from about a minute to several hours, and some rules were capped at 100 matching documents per run.
- Huntress says maintaining data redundancy in Elasticsearch increased costs, while limited node customization and inter-availability-zone transfer charges added expense.
- The new design sends processed documents to both hosted OpenSearch for search and an event stream consumed by multiple instances of a custom detection engine.
- Huntress says message queuing preserves its at-least-once processing guarantee, while streaming removes scheduled-query delays and the 100-result cap.
Article Details
- Topic
- Huntress's transition to a custom detection engine and AWS-hosted OpenSearch to improve detection speed, scalability, and cost efficiency.
Vendors
Amazon Web ServicesRead on to learn how we transformed Huntress’ data and log analysis by transitioning to a custom detection engine, leveraging OpenSearch hosted by AWS.Elasticdatabase before sending them to Elasticsearch. The data within the survey file loosely follows the Elastic Common Schema format to eliminate the need to transform them before Elasticsearch indexing.HuntressHomeBlogHow Huntress Transformed Its Detection Engine
Products
Elastic StackOnce the documents are in Elasticsearch, they are available to other tools within the Elastic Stack and our Security Operations Center (SOC) analysts. This interoperability enables us to utilize the Security app withinElasticsearchthen enriches the documents and stores key fields in our relational database before sending them to Elasticsearch. The data within the survey file loosely follows the Elastic Common Schema format to eliminate theKibanaOperations Center (SOC) analysts. This interoperability enables us to utilize the Security app within Kibana to execute the detection rules. These rules are written in the Sigma format and then converted to aOpenSearchRead on to learn how we transformed Huntress’ data and log analysis by transitioning to a custom detection engine, leveraging OpenSearch hosted by AWS.