Huntress Explains Why AI-Generated Security Work Still Needs Human Checks
Summary
Huntress argues that AI can help build and analyze security detections, but people must define and run checks to catch faulty conclusions, measure false positives, and decide whether rules are safe to ship.
Key points
- An AI model mistook a legitimate Microsoft Outlook client identifier for malicious tooling because it judged the identifier against too narrow a baseline.
- A broader comparison surfaced a malicious identifier seen in only three organizations; it was used to rewrite inbox rules and now informs a live detection with two true positives and no false positives so far.
- The author recommends defining checks—such as testing against known-good and known-bad data, measuring alert volume, and checking for overlap with existing rules—before delegating work.
- AI can reduce the effort needed to build analysis tools and detection rules, but people still need to set scope, evaluate results, and make deployment decisions.
- The article warns that rubber-stamping a check can let faulty detections reach production, while handing off all practical work may weaken a person's ability to spot subtle errors.
Article Details
- Topic
- Using verification checks and iterative workflows to safely delegate detection engineering and other work to AI
People
Andrej KarpathyCredited with calling the fast generate-check-refine process the “generation-verification loop.”Brian ConcannonQuoted on the importance of testing detection rules against known-good and known-bad data before release.Dan ShipperCited for writing about judgment and taste becoming more valuable as AI makes competent output easier to produce.Kieran KlaassenCredited with describing a human-machine-human workflow as the “AI sandwich.”
Products
Microsoft 365 Apps for EnterpriseIf an attacker uses a stolen Microsoft 365 session to take over an account without tripping any detections, the question is what should have caught it: an existing rule that was too precise, or one that doesn't existMicrosoft Outlookin a known-compromised mailbox and was uncommon in that organization. But the identifier was the Microsoft Outlook desktop client, which shows up in legitimate activity across more than 17% of our customer