BSides NYC Talk Explains How to Collect and Correlate AI Audit Logs

· Original article ↗

Summary

A BSides NYC presentation outlined what AI platform, runtime, transcript, and endpoint logs capture, how to correlate events, and why rejected tool calls matter. Some client telemetry must be enabled in advance.

Key points

  • AI agents may access local files, run commands, and call cloud or SaaS APIs using available permissions and credentials.
  • Platform audit logs capture account, API key, and policy changes; runtime events record tool calls and approval decisions; transcripts contain prompts and responses; endpoint logs help confirm executed commands and file changes.
  • Collection requirements vary by tool, and client telemetry must be enabled before activity occurs; missing events cannot be recovered afterward.
  • In Claude Code, session.id groups events by session, prompt.id groups events by prompt, and tool_use_id links a tool call’s approval decision to its result.
  • Claude Code records rejected calls in tool_decision without a corresponding tool_result, so searches limited to tool results can miss rejected attempts.
  • The presentation compared logs from an approved, rejected, and automatically handled attempt to copy a fake credentials file.
  • The author said Claude Code added classifier_verdict and permission_mode fields to OpenTelemetry tool_decision events the morning after the demo scenarios.

Article Details

Defense Focus
Improve detection, response, and investigation of AI-agent activity by understanding and correlating audit logs, runtime events, transcripts, and endpoint telemetry.
Detection Methods
  • Correlate Claude Code events using session.id for sessions, prompt.id for prompts, and tool_use_id to link tool-call approval decisions with results.
  • Inspect tool_decision events for rejected calls, including events without a corresponding tool_result.
  • Compare logs generated by approved, rejected, and automatically handled tool-call scenarios.
  • Use endpoint logs to confirm commands executed and files changed.
Data Sources
  • Platform audit logs covering account, API key, and policy changes
  • Runtime events covering tool calls and approval decisions
  • AI interaction transcripts containing prompts and responses
  • Endpoint logs showing executed commands and changed files
  • Claude Code OpenTelemetry tool_decision events, including classifier_verdict and permission_mode
  • Gemini for Workspace activity logs showing invocation source and user; prompts and responses are not included
Platforms
  • Claude Code
  • Gemini for Workspace
Defensive Actions
  • Enable required client telemetry before relevant activity occurs; missing historical events cannot be recovered by enabling it afterward.
  • Verify the plans, permissions, and settings required to collect logs for each tool.
  • Include rejected tool_decision events in searches rather than relying only on tool_result events.
  • Account for differences in logged fields and behavior across providers and tools when designing collection and investigations.

People

Products

Related Articles