Researchers Find 336,000 Internet-Exposed Prometheus Servers and Exporters
Summary
Aqua Security researchers found about 336,000 internet-exposed Prometheus servers and exporters, which could leak sensitive data, enable denial-of-service attacks through pprof endpoints, or expose users to malicious exporters via RepoJacking.
Key points
- Shodan searches identified more than 296,000 exposed exporters and 40,000 Prometheus servers.
- Unauthenticated servers and metrics endpoints could expose credentials, API keys, internal endpoints, subdomains, and infrastructure details.
- Researchers demonstrated that requests to exposed /debug/pprof endpoints could exhaust resources, disrupt hosts, or repeatedly crash Kubernetes pods.
- The researchers identified RepoJacking risks in several exporters linked from Prometheus documentation, potentially allowing malicious code to reach users who deploy them.
- Skoda blocked access to an exposed Prometheus instance after the researchers reported it.
- A pull request addressing the exporter-link risk was merged; the article says configurable controls for separating or disabling endpoints were still on the Prometheus team's backlog.
- Recommended mitigations include authentication, limiting public exposure, restricting profiling endpoints, setting resource limits, and verifying project links.
Article Details
- Attack Vectors
- Unauthenticated internet-facing Prometheus servers and exporters can be queried to disclose metrics, credentials, passwords, authentication tokens, API keys, internal endpoints, subdomains, and infrastructure details.
- Repeated simultaneous requests to exposed /debug/pprof endpoints, including /debug/pprof/heap?seconds={i}, can consume CPU and memory and cause host or Kubernetes pod instability or outages.
- RepoJacking can enable remote code execution when an attacker claims a deleted or renamed GitHub account or repository name still referenced by exporter links, then supplies a malicious exporter that users clone and deploy.
- Defensive Notes
- Protect Prometheus servers and exporters with authentication and authorization, and minimize their exposure to the public internet; use VPNs or private networks where external access is necessary.
- Restrict /debug/pprof and other debugging or profiling endpoints to internal use or disable them in production.
- Apply CPU and RAM resource limits to reduce the impact of resource-exhaustion attacks.
- Verify that open-source project links point to the correct original repositories before downloading or deploying code.
CVE
People
Andrey PolkovnychenkoNamed as a JFrog researcher whose prior report highlighted information-disclosure risks from publicly accessible Prometheus servers or exporters.David de Torres HuertaNamed as a Sysdig researcher whose prior report highlighted information-disclosure risks from publicly accessible Prometheus servers or exporters.Miguel HernándezNamed as a Sysdig researcher whose prior report highlighted information-disclosure risks from publicly accessible Prometheus servers or exporters.Shachar MenasheNamed as a JFrog researcher whose prior report highlighted information-disclosure risks from publicly accessible Prometheus servers or exporters.
Vendors
Products
AlertmanagerThe system is modular, with components such as Alertmanager for handling alerts and Grafana for visualization, making it ideal for dynamic environments like Kubernetes.AWS Cost ExporterAWS Cost Exporter AWS EC2In cloud-hosted environments, such as AWS EC2 instances, our tests demonstrated that a DoS attack caused a loss of communication with the host, requiring a machine reboot to restore functionality.Azure Cost ExporterAzure Cost Exporter Docker Cloud ExporterDocker Cloud Exporter Docker Hub ExporterDocker Hub Exporter GrafanaThe system is modular, with components such as Alertmanager for handling alerts and Grafana for visualization, making it ideal for dynamic environments like Kubernetes.KubernetesAdditionally, we identified an alarming risk of DoS attacks stemming from the exposure of pprof debugging endpoints, which, when exploited, could overwhelm and crash Prometheus servers, Kubernetes pods and other hosts.Kubernetes Cost ExporterKubernetes Cost Exporter Nginx VTS ExporterNginx VTS Exporter Node ExporterWe observed another example of information disclosure through an exposed /metrics endpoint of Node Exporter.PrometheusIn this research, we uncovered several vulnerabilities and security flaws within the Prometheus ecosystem.Rancher ExporterRancher Exporter Tinyproxy ExporterTinyproxy Exporter
Tools
pprofAdditionally, we identified an alarming risk of DoS attacks stemming from the exposure of pprof debugging endpoints, which, when exploited, could overwhelm and crash Prometheus servers, Kubernetes pods and other hosts.ShodanWe used Shodan, a search engine designed to identify internet-connected devices, to analyze the presence of publicly accessible Prometheus servers and exporters.