How to Export Logs from OpenSearch
Summary
A practical guide to exporting OpenSearch logs with Python or curl, including permissions, field selection, pagination limits, and troubleshooting for security-log review.
Key points
- Check an example document and index mapping to identify the time field and original log field before querying.
- With the security plugin enabled, the user needs read and search/scroll permissions; Amazon OpenSearch Service may require IAM credentials with SigV4 signing.
- For larger exports, the guide recommends Python’s opensearch-py helpers.scan; it also describes using a point in time with search_after.
- A single curl request can retrieve up to 10,000 documents by default; larger requests exceed the default result window.
- Export only the fields needed, such as the timestamp, host, and raw message, and review the file before uploading it to LogTotal.
- The troubleshooting section covers access errors, oversized result windows, empty results, expired scroll contexts, and Amazon OpenSearch Service signature errors.
Article Details
- Defense Focus
- Export and review OpenSearch logs for investigation, with guidance on selecting relevant fields and handling large exports.
- Detection Methods
- Query log indices with a time-range filter and inspect the original log line from message, event.original, or Wazuh full_log.
- Check the index mapping and run a count query before exporting to verify the selected field and result volume.
- Page through large result sets with helpers.scan or use a point in time with search_after.
- Data Sources
- OpenSearch documents in logs-* indices, including @timestamp or timestamp and original-log fields such as message or event.original.
- Wazuh alerts in wazuh-alerts-* and original log lines in full_log.
- Rule Types
- OpenSearch Query DSL
- Platforms
- OpenSearch
- LogTotal
- Defensive Actions
- Use an account with read access to the relevant index pattern and search/scroll permissions.
- Limit exports to the needed time range and fields to reduce file size.
- Review and sanitize exported logs before uploading them to LogTotal.
- For exports above 10,000 results, use helpers.scan or a point in time with search_after rather than a single search request.
- Use explicit UTC timestamps and verify the mapped time field when results are unexpectedly empty.
- For Amazon OpenSearch Service with IAM-based access, sign requests with SigV4.
Vendors
Products
Amazon OpenSearch ServiceOpenSearch backs a lot of homegrown SIEMs and log platforms, including Amazon OpenSearch Service and the Wazuh indexer.LogTotalOpenSearch Dashboards can generate reports from saved searches, but not in a format LogTotal reads today.OpenSearchOpenSearch Dashboards can generate reports from saved searches, but not in a format LogTotal reads today.OpenSearch DashboardsOpenSearch Dashboards can generate reports from saved searches, but not in a format LogTotal reads today.Wazuh indexerOpenSearch backs a lot of homegrown SIEMs and log platforms, including Amazon OpenSearch Service and the Wazuh indexer.