cyber.hendryadrian.com
Security Report · sentinelone.com

Study Evaluates LLMs for Turning Cyber Threat Reports Into Knowledge Graphs

· Original article ↗

Summary

A preliminary evaluation finds LLM-based CTI extraction can cut report processing from 41 minutes to about 3.3 minutes, but accuracy and coverage vary with report structure, evidence ambiguity, prompts, and model choice.

Key points

  • The proposed pipeline sanitizes CTI reports, uses specialized LLM extractors for infrastructure, executables, and playbooks, then assembles linked information into a knowledge graph.
  • The evaluation tested GPT-4.1, GPT-5, GPT-5.2, Claude Sonnet 4.5, and Claude Opus 4.5 on 343 atomic IOCs and 1,859 labeled IOC attribute instances; the authors describe the results as preliminary, not a definitive model ranking.
  • LLM processing averaged about 3.3 minutes per report versus 41 minutes for human analysts, a reported speedup of more than 18 times, with trade-offs in completeness and correctness.
  • Extraction quality varied with report formatting, contextual labels, OCR coverage, prompt-model fit, and data-model wording.
  • The workflow uses evidence grades and an explicit abstention option; the authors recommend representative ground truth that can account for genuine ambiguity.
  • Ensemble benefits differed by attribute, while errors and omissions in reconstructed playbooks and graphs could affect downstream analysis; the article recommends ongoing evaluation and safeguards.

Article Details

Publisher
SentinelOne
Scope
Preliminary evaluation of LLM-based extraction and knowledge-graph construction from cyber threat intelligence reports.
Sample Size
Ground truth of 343 atomic IOCs and 1,859 labeled IOC contextual attribute instances.
Key Statistics
  • Human analysts averaged 41 minutes per report; the LLM extractors averaged approximately 3.3 minutes per report, with a reported aggregate speed-up of more than 18 times.
  • The slowest LLM-based setup was approximately six times faster than the human baseline; the fastest reduced average processing time by more than 97%.
  • Playbook and knowledge-graph assembly was evaluated using 17 structural and semantic metrics.
Recommendations
  • Define evidence standards and output-quality thresholds appropriate to the intended downstream use, with stricter thresholds for mission-critical automation.
  • Use representative ground truth that permits multiple defensible labels for genuinely ambiguous evidence.
  • Monitor abstention and speculative-assignment rates as input data changes.
  • Evaluate LLM ensembles for specific extraction tasks or attributes rather than applying them uniformly.
  • Apply safeguards such as generation guardrails or automated consistency checks to critical knowledge-graph relationships.

Vendors

Anthropicof an evaluation study of several off‑the‑shelf, general‑purpose language models from OpenAI and Anthropic — GPT‑4.1, GPT‑5, GPT‑5.2, Claude Sonnet 4.5, and Claude Opus 4.5 — used within theOpenAIthe results of an evaluation study of several off‑the‑shelf, general‑purpose language models from OpenAI and Anthropic — GPT‑4.1, GPT‑5, GPT‑5.2, Claude Sonnet 4.5, and Claude Opus 4.5 — used within the

Products

Claude Opus 4.5Evaluates off‑the‑shelf LLMs (GPT‑4.1, GPT‑5, GPT‑5.2, Claude Sonnet 4.5, Claude Opus 4.5) on a ground truth of 343 atomic IOCs and 1,859 labeled IOC attribute instances, emphasizing feasibility rather thanClaude Sonnet 4.5Evaluates off‑the‑shelf LLMs (GPT‑4.1, GPT‑5, GPT‑5.2, Claude Sonnet 4.5, Claude Opus 4.5) on a ground truth of 343 atomic IOCs and 1,859 labeled IOC attribute instances, emphasizing feasibility rather than

Related Articles