Railway security incident: environment variables may be exposed. A practical secret rotation guide

What happened at Railway

Railway is a platform-as-a-service where applications read their configuration, including API keys and database passwords, from environment variables that Railway stores encrypted. Railway’s preliminary report describes the attack:

• A probable privilege escalation in its build infrastructure.
• Access to CI credentials, which let the attacker decrypt artifacts holding further API access.
• An unauthorised query of encrypted environment variables, a subset of which was decrypted with the compromised credentials.

Railway has not published how the attacker first got in, and the forensic investigation is still open. All times below are UTC.

Running services stayed online. Only builds and deployments were paused.

What was exposed, and who is at risk

• Scope: “at most 1.85% of all workspaces had some environment variables potentially compromised”, about 100,000 workspaces. Because the variables use envelope encryption, Railway can’t identify exactly which keys were decrypted.
• Highest risk: workspaces with a very large number of environment variables, and environments holding many AI provider keys.
• The target: AI provider keys, to run model distillation attacks, according to Railway. It also calls the incident part of a series of attacks on platform companies.
• Not affected: Railway user accounts, according to Railway. It has not ruled out access to customer source code and will confirm in its final report.
• Everyone else: Railway e-mailed definite-impact and high-risk workspaces, but says the others are “not confirmed safe from compromise”.

Railway has published no indicators of compromise and has not confirmed misuse of stolen secrets. The Hacker News reports a third-party claim of crypto wallets drained after their private keys were stored on Railway. Railway has not confirmed it.

What Railway did

• Halted builds, shut down potentially compromised internal tools, and rebuilt them from validated source.
• Rolled all its internal keys twice and rotated the credentials for all storage buckets.
• Invalidated every API and project token and every user session, and rolled all GitHub App installations.
• Moved builds to single-tenant builders.
• Asked GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe, Google and others to revoke keys. It gave GitHub and SpaceXSI every key present on the platform, so a revocation notice from a provider does not by itself mean your key was decrypted.

How to rotate your secrets, in order

Railway’s request is broad: revoke and replace every sensitive variable stored before 21 October, including private registry credentials, and redeploy. With dozens of secrets, the order matters. Start with what an attacker can turn into money or wider access fastest.

1. Make the inventory

Export the variable names of every project and environment (Railway’s dashboard or railway variables with a new token). Mark each one with its provider, what it grants and who can rotate it. Shared variables and reference variables count too.

2. Rotate in this order

3. Rotate without an outage

1. Create the new credential at the provider, with the narrowest scope and an expiry or spending cap where the provider supports it.
2. Store it on Railway (or move it to a secrets manager) and redeploy. Railway invalidated old API tokens, so create a new one for the CLI and for CI.
3. Check the service works with the new value: health checks, a test request, the provider’s “last used” timestamp.
4. Revoke the old credential at the provider. A secret that is replaced but never revoked is still usable by whoever stole it.
5. Write down what you rotated and when. You will need it to read logs and to answer Railway or your own customers.

How to spot misuse

• AI providers: usage and cost dashboards from 20 October onwards. Look for spikes, models you don’t use and requests from unknown IP addresses or regions. Railway has published checks for OpenAI, Anthropic, OpenRouter and Grok usage.
• AWS and GCP: CloudTrail and Cloud Audit Logs for calls made with your keys from outside your infrastructure. Start with GetCallerIdentity, new IAM users, keys or roles, and unexpected regions.
• GitHub and GitLab: the organisation audit log for new tokens, deploy keys, OAuth or GitHub App authorisations and repository clones. Also review which apps have access.
• Registries: Docker Hub, npm and PyPI audit logs for pushes or publishes you didn’t make.
• Stripe and payment providers: API request logs and events for refunds, payouts, new webhooks or keys created after the incident.
• Databases: connections from unknown addresses, new users and large exports, especially on databases that had public network access.

The absence of alerts proves little. Many providers only keep detailed logs for a short time, so export them now.

Stop storing long-lived keys

Rotation fixes this incident. Short-lived credentials make the next one smaller:

• Use workload identity federation (OIDC) instead of cloud access keys. GitHub Actions, GitLab CI and many platforms can exchange a signed identity token for temporary AWS, GCP or Azure credentials that expire in minutes, so there is no key to steal from an environment variable.
• Keep secrets in a secrets manager with dynamic credentials (such as Vault or your cloud’s secret store). The application then fetches database credentials that expire, instead of carrying a password for years.
• Scope and cap API keys. Use one key per service and environment, give it only the permissions it needs, set spending limits and IP allowlists where providers support them, and give it an owner and an expiry date.
• Make rotation routine. A service that can’t rotate a secret without downtime will put it off during an incident. Rehearse it.

Where OpenWAF fits

A WAF does not protect secrets stored inside a hosting platform, and it can’t stop an attacker from using a stolen API key directly against the provider. What it can do is limit what a stolen application secret gets someone through your front door, and keep your origin from being reached around it:

• An origin that only answers OpenWAF. With authenticated origin pulls (mTLS), the OpenWAF edge presents a client certificate and your server can refuse every connection that doesn’t come through OpenWAF. Someone who learns your origin’s address from leaked configuration still can’t skip your WAF rules, rate limits or access gate. The client CA is shared across the platform: it proves the connection comes from the OpenWAF edge, not that it is for your zone, so keep checking Host as well.
• An access gate that doesn’t depend on app secrets. If a session or JWT signing secret leaked, an attacker can forge a valid-looking admin session. The access gate holds admin paths such as /admin or /wp-admin at the edge until the person enters a one-time code sent to an authorised e-mail address. A forged cookie alone is not enough.
• Rate limits and IP rules on login, password reset and API routes slow down anyone testing stolen credentials (rate limiting, IP and country rules).
• Rotate your OpenWAF token too. If an OpenWAF API token was among your Railway variables, revoke it in the dashboard and create a new one. Tokens can be read-only or limited to one client or one domain, and every change made with a token is versioned and in the audit log.

OpenWAF protects HTTP and HTTPS traffic. It does not sit in front of database ports, so closing public database access, as Railway recommends, is still on you.

Checklist

1. Export the names of every Railway variable, in every project and environment, set before 21 October 2026.
2. Rotate AI provider keys, cloud credentials and wallet keys first, then payments, source control and registries, databases, application secrets and other SaaS keys.
3. For each one: create new, deploy, verify, then revoke the old one at the provider.
4. Create new Railway API and project tokens and update CI and scripts.
5. Remove public network access from databases that don’t need it.
6. Review provider usage, CloudTrail or Cloud Audit Logs, GitHub, registry, Stripe and database logs from 20 October, and export them before they expire.
7. Replace long-lived cloud keys with OIDC federation and short-lived credentials, and add spending caps and expiries to API keys.
8. Lock your origins to your WAF with mTLS, and put admin areas behind an access gate that doesn’t rely on application secrets.

Frequently asked questions

Sources

• Security Disclosure: Wednesday, October 21st Railway, Oct 21, 2026
• Preliminary report: October security incident Railway, Oct 22, 2026
• Railway Platform Outage (incident QH8L8B3E) Railway Status, Oct 21, 2026
• Security Disclosure Recommendation: Check for Anomalous AWS Access Railway, Oct 22, 2026
• Security Disclosure Recommendation: Check for Unexpected OpenAI Usage Railway
• Railway says intruder may have accessed customer environment variables The Hacker News, Oct 22, 2026