Elastic Details Framework for Benchmarking LLMs in Agentic SOC Workflows
Summary
Elastic describes a benchmark for agentic SOC models that grades tool use and grounded workflow execution across security tasks, alongside Attack Discovery and SIEM-rule migration; it says model rankings vary by capability.
Key points
- Elastic’s framework evaluates whether models select appropriate security skills and tools, execute workflows, and ground answers in tool results rather than relying on generic LLM benchmark scores.
- The agent suite tests seven capabilities with 21 prompts and records tool calls, parameters, results, latency, and token usage.
- A synthetic Chrysalis DLL-side-loading scenario provides repeatable alert and endpoint data; its loader uses the harmless EICAR test hash for a stable VirusTotal verdict.
- Scoring is blind to model identity and evaluates answer quality alongside tool and skill selection; answers without required tool calls receive limited credit.
- Two additional suites evaluate Attack Discovery using roughly 95 alerts covering eight intrusions, and Automatic Migration using 52 Splunk detection rules and 200 macros.
- Elastic reports that model rankings vary across capabilities, arguing that model selection for an agentic SOC should depend on the task.
Article Details
- Topic
- Evidence-based evaluation framework for agentic SOC LLMs across security workflows