How to Export Elasticsearch and Kibana Logs for Security Investigations
Summary
A practical guide to exporting Elasticsearch logs with Python, ES|QL, or Kibana Discover, including API-key permissions, format and row limits, and preparing log records for investigation in LogTotal.
Key points
- Create an API key with read and view_index_metadata permissions for the relevant indices.
- For larger exports, use Python’s helpers.scan or a point-in-time search with search_after; a single _search can hit the result-window limit.
- ES|QL exports default to 1,000 rows and support up to 10,000; its JSON output uses columns and values rather than one record per line.
- Kibana Discover can export filtered results as CSV or copy selected documents as JSON, with limits that vary by version and configuration.
- Check whether event.original is available to retain raw log lines; otherwise, export the parsed fields.
- The guide recommends narrowing time ranges for older data and describes common fixes for permission errors, row limits, missing fields, and timeouts.
Article Details
- Defense Focus
- Export and prepare Elasticsearch logs for security investigation in LogTotal.
- Detection Methods
- Filter authentication activity in Kibana Discover using event.category: authentication.
- Scope Elasticsearch searches to a relevant time range and select fields such as @timestamp, host.name, user.name, source.ip, and message.
- Include event.original when available to retain the raw log line.
- Data Sources
- Elasticsearch indices and data streams, including logs-system.auth-* and logs-<dataset>-<namespace>.
- Linux authentication logs from the Elastic System integration.
- Parsed event fields and, when preserved, event.original.
- Rule Types
- ES|QL queries
- Elasticsearch Query DSL
- Kibana Discover search and filters
- Defensive Actions
- Create an API key with read and view_index_metadata privileges for the required indices.
- Use helpers.scan or point-in-time searches with search_after for large exports; use ES|QL LIMIT for quick pulls of up to 10,000 rows.
- Keep time windows narrow, especially when searching cold or frozen data tiers.
- Check whether event.original is preserved; if not, export the parsed document.
- Review and sanitize exported logs before uploading them, and convert records to JSONL when that format is required.
Vendors
Products
Elastic AgentElastic Agent integrations write to data streams named logs-<dataset>-<namespace>.Elastic SecurityElastic is one of the most common foundations for in-house security analytics, from Elastic Security to self-built stacks.ElasticsearchOn Elastic Cloud, the Elasticsearch endpoint is on your deployment’s overview page.KibanaFor quick pulls of up to 10,000 rows, ES|QL with format=json works straight from Kibana’s Dev Tools.LogTotalKibana Discover can generate a report too, but not in a format LogTotal reads today.