Railway Incident Highlights CI/CD Risks and Runtime Defenses

· Original article ↗

Summary

Railway reported that an attacker partially escaped build isolation and later used a separate OIDC configuration error to access systems holding customer variables. StepSecurity outlines CI/CD isolation, access-control and runtime-monitoring defenses.

Key points

  • Railway reported that an attacker exploited a vulnerability to partially escape filesystem isolation, read host VM files and obtain a build infrastructure credential.
  • According to Railway, misplaced trust in that credential enabled access to build histories and internal source; a separate OIDC configuration error later enabled cloud access.
  • Railway said at most 1.85% of workspaces—about 100,000—could have had customer variables decrypted; this is an upper bound.
  • The article recommends disposable, isolated build environments; separating untrusted builds from privileged deployments; and limiting job credentials and internal network access.
  • It advises auditing outbound connections before enforcing an allowlist, and testing OIDC trust conditions, role permissions and protected-environment approvals.
  • StepSecurity says Harden-Runner provides workflow-context network monitoring and, on supported configurations, process or file-write visibility and outbound controls; capabilities vary by platform and plan.
  • StepSecurity’s analysis of the separate tj-actions/changed-files compromise reported an unexpected endpoint and exposed secrets in build logs.

Article Details

Defense Focus
Protect CI/CD build and deployment jobs through runtime visibility, restricted outbound access, workload isolation, and narrowly scoped cloud identity permissions.
Detection Methods
  • Correlate network connections with the responsible process and workflow step.
  • Compare job network activity with baselines from previous runs and investigate unexpected destinations.
  • Use file-write visibility and source-code overwrite detections to identify build-time changes, where supported.
  • Test egress controls by attempting a connection to an unapproved destination and confirming it is blocked and recorded.
  • Review OIDC token claims and verify that jobs outside the protected environment cannot obtain the production cloud role.
Data Sources
  • Runner network events, including destination, port, and workflow-step context.
  • Process events, including process names and arguments, where supported.
  • File-write events, including affected paths and responsible processes, on supported configurations.
  • Workflow activity and network baselines from previous runs.
  • OIDC token issuer, audience, and subject claims.
  • Runner and infrastructure logs retained after runner pods are deleted.
Rule Types
  • Outbound egress allowlists with audit and block modes.
  • Network-baseline anomaly detection for unexpected destinations.
  • Known-malicious-destination blocklists.
  • Source-code overwrite detections.
  • OIDC trust-policy conditions tied to the intended repository and protected environment.
Platforms
  • Self-hosted VM runners
  • Ephemeral runner pods
  • Kubernetes DaemonSet deployments
Defensive Actions
  • Separate untrusted build jobs from privileged deployment jobs; use isolated, disposable execution environments and avoid sharing writable workspaces or sensitive host directories.
  • Keep production credentials out of jobs that do not need them, and limit each build credential to the resources and operations required.
  • Restrict runner access to internal services, host resources, container runtimes, Kubernetes APIs, and secrets; review authorization at receiving services as well as at runners.
  • Review representative job runs in egress audit mode, validate required destinations, then enforce an explicit allowlist on supported runners.
  • Grant OIDC token-writing permission only to jobs that need it; bind production roles to the intended repository and protected environment, and require approval for sensitive deployments.
  • Move unrelated checks and third-party code out of jobs that assume production roles; use separate, less-privileged jobs for preparation and file-change evaluation.
  • Patch runner images and infrastructure, retain logs, and test that unapproved jobs and destinations are denied.
  • Route blocked connections and unexpected activity to the teams responsible for the workflow.

MITRE ATT&CK

Vendors

Products

Tools

Related Articles