Wiz Shares Cloud Persistence-Hunting Guidance Following CircleCI Breach

Jan 12, 2023 · Original article ↗

Summary

Following CircleCI’s report that its breach exposed sensitive data and secrets belonging to thousands of organizations, Wiz outlines AWS, Azure, and GCP persistence techniques, log queries, and remediation steps, later adding queries based on CircleCI IOCs.

Key points

CircleCI reported that its development-platform breach exposed sensitive data and secrets belonging to thousands of organizations; it urged users to rotate stored secrets and review internal logs.
Wiz cautions that secret rotation alone may not remove attacker access if cloud persistence was established.
The guide identifies potentially suspicious AWS activity, including creating users or keys, changing login profiles, launching instances, modifying startup data, and altering Lambda functions.
For Azure, it highlights service-principal secrets, new users or password resets, Automation runbooks and webhooks, VM changes, and function-code updates.
For GCP, it covers new service-account keys, function changes, VM access or creation, and startup scripts.
The article provides queries for AWS Athena, Azure Log Analytics, and GCP Logs Explorer, plus investigation and remediation steps for suspicious findings.
An update adds queries to search cloud logs for IP addresses CircleCI identified as IOCs and recommends investigating matches and checking for persistence or data exfiltration.

Article Details

Defense Focus: Detect and investigate cloud persistence established with compromised credentials, including after secrets have been rotated, with additional hunting for activity from IPs identified as malicious in the CircleCI incident.
Detection Methods: Search cloud audit and activity logs for suspicious identity, credential, compute, serverless, automation, and remote-command operations during 2022-12-16 through 2023-01-04.
Use suspicious AWS access key IDs to find persistence-related API calls, repeated failed AssumeRole attempts across more than five distinct roles, and subsequent successful role assumptions.
Use suspicious Azure service-principal object or client IDs to hunt for runbook, webhook, VM, function-code, application-credential, user-creation, and password-reset activity.
Use suspicious GCP service-account details to search logs for key creation, function changes, startup-script changes, and VM activity.
Search AWS, Azure, Azure service-principal sign-in, and GCP logs for the eight IPs listed as known malicious in the article’s CircleCI appendix.
Data Sources: AWS CloudTrail logs, including source IP, event name, error code, access key, principal, and role information.
AzureActivity logs, including caller, caller IP, operation name, and activity status.
Azure AuditLogs and AADServicePrincipalSignInLogs, including application activity, result, and IP address.
GCP Cloud Logging entries, including resource type, method name, caller IP, principal email, severity, and instance metadata changes.
Rule Types: SQL queries for AWS Athena.
KQL queries for Azure Log Analytics.
Google Cloud Logging query for Logs Explorer.
Platforms: AWS cloud accounts and workloads.
Azure cloud tenants and workloads.
GCP cloud projects and workloads.
Defensive Actions: Rotate or remove exposed secrets and keys, but continue hunting for persistence regardless of rotation.
Investigate suspicious query results; remove unfamiliar users, access keys, secrets, certificates, and webhooks, and revoke suspicious role-session permissions.
Change passwords for affected users and enforce MFA for login profiles and accounts.
Remove suspicious compute instances or key pairs; revert unauthorized instance or startup-script changes and inspect workloads for backdoors.
Restore modified serverless-function code and remove unfamiliar functions or automation runbooks.
Back up important data before deleting or terminating affected machines, and investigate for signs of data exfiltration.

Indicators of compromise

Type	Indicator	Context
IPV4	`111[.]90[.]149[.]55`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`178[.]249[.]214[.]10`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`178[.]249[.]214[.]25`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`188[.]68[.]229[.]52`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`72[.]18[.]132[.]58`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`89[.]36[.]78[.]109`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`89[.]36[.]78[.]135`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.
IPV4	`89[.]36[.]78[.]75`	Listed by the article as a known malicious IP from the CircleCI incident for hunting cloud activity.