ChaosDB Research Details Exploit Chain Exposing Azure Cosmos DB Customer Keys
Summary
Wiz details how ChaosDB chained Jupyter Notebook misconfigurations to expose Azure Cosmos DB customer keys and internal infrastructure. Microsoft disabled the vulnerable feature in August 2021; researchers observed some credentials being revoked.
Key points
- Wiz reports demonstrating unauthorized access to several thousand customers’ Cosmos DB environments through a chain disclosed to Microsoft in August 2021.
- The Jupyter Notebook C# host ran as root, allowing researchers to elevate privileges and remove container-local firewall rules blocking Azure metadata services and internal networks.
- Researchers queried WireServer and reverse-engineered the Windows Azure Guest Agent’s Certificates Bond Package handling, recovering 25 Microsoft certificates and corresponding private keys.
- Recovered certificates enabled Service Fabric authentication and decryption of other tenants’ database primary keys, notebook authentication tokens and notebook storage credentials.
- Researchers also accessed internal storage containing infrastructure metadata and operational logs, and authenticated to more than 100 internet-accessible Service Fabric instances across multiple regions using one certificate.
- Wiz observed the vulnerable feature disabled on August 14, 2021, and some obtained credentials revoked on August 16. According to its timeline, Microsoft confirmed several thousand affected customers on August 23.
Article Details
- Attack Vectors
- Wiz researchers executed C# code through the embedded notebook host, which ran as root, then added a UID 0/GID 0 account to /etc/passwd and switched to it.
- Root access allowed researchers to flush container-local firewall rules and reach previously blocked host metadata, WireServer, and internal network endpoints.
- WireServer accepted a researcher-supplied transport certificate and returned a certificate bundle containing 25 Microsoft certificates and their corresponding private keys.
- Recovered certificates enabled authentication to Service Fabric and decryption of extension settings and application secrets, exposing other customers' database keys, notebook authentication tokens, and notebook storage credentials.
- Researchers found more than 100 internet-accessible Service Fabric instances on port 19080 and reported successful authentication across multiple regions using one recovered certificate.
- Defensive Notes
- Wiz recommended enforcing network restrictions outside the Jupyter Notebook container so that root access inside the container cannot bypass them.
- Wiz observed that the vulnerable feature had been disabled by 2021-08-14 and that some recovered credentials had been revoked by 2021-08-16; the article does not establish that every exposed credential was revoked.
MITRE ATT&CK
T1046 · Network Service DiscoveryResearchers scanned Microsoft's ASN for port 19080 and found more than 100 externally accessible Service Fabric instances.T1059.004 · Unix ShellResearchers used the notebook's Linux terminal to run commands including su, iptables, curl, and certificate-processing commands.T1082 · System Information DiscoveryResearchers queried IMDS to identify the underlying host's operating system, subscription, VM configuration, and network information.T1136.001 · Local AccountResearchers appended an account with UID 0 and GID 0 to /etc/passwd inside the notebook container and switched to it using su.T1140 · Deobfuscate/Decode Files or InformationResearchers decoded Certificates Bond Package data and used OpenSSL with recovered private keys to decrypt extension protectedSettings and Cosmos DB application secrets.T1526 · Cloud Service DiscoveryResearchers used sfctl application list to enumerate more than 500 Cosmos DB application instances managed by a regional Service Fabric cluster, including instances outside their account.T1552.001 · Credentials In FilesExtension configuration and Service Fabric application listings exposed encrypted storage keys, database keys, and notebook secrets that researchers recovered using the obtained certificates.T1552.004 · Private KeysResearchers retrieved and decoded a WireServer certificate bundle containing 25 Microsoft certificates and their corresponding private keys.T1562.004 · Disable or Modify System FirewallResearchers ran iptables -F as root to remove container-local firewall restrictions blocking metadata services and the internal subnet.
People
Vendors
Products
Azure Cosmos DBSet up a Jupyter Notebook container on your Azure Cosmos DBAzure Service Fabrictime to pause and ask, what is the Service Fabric anyway? According to Microsoft’s documentation, Azure Service Fabric is a distributed systems platform that makes it easy to package, deploy, and manage scalable andAzure Storage ExplorerWe accessed these storage accounts using Azure Storage Explorer and found hundreds of gigabytes of metadata and operational logs, as well as millions of records about Cosmos DB’s underlying infrastructure:Jupyter NotebookSet up a Jupyter Notebook container on your Azure Cosmos DBWindows Azure Guest Agent.exeUnlike the WA-Agent, the Windows virtual machine agent (also known as the Windows Azure Guest Agent.exe), is not open-source. But it is written in C#, so we could decompile it into something that resembles source code
Tools
ILSpyresembles source code fairly easily. There are a number of tools out there that can do this—we chose ILSpy.OpenSSLThis is the first time we’d ever encountered this format. Unfortunately, the OpenSSL commands that we are used to executing to decrypt the standard format did not work here. Time to up our game and try to decode thissfctlto our local Service Fabric on port 19080 with the fabric.westus1.cosmos.azure.com certificate, using the sfctl command line. We then used the sfctl application list command in order to list the running application