Short answer: assume every value stored on the platform is known to someone else and list all of them. Replace them in order of what a thief could do with each one: AI provider keys and cloud credentials first, then payment keys, then private keys and database credentials. For each secret create the new value at the issuer, deploy it, check that it works, and revoke the old one. After a breach, keep the gap between "new value live" and "old value revoked" short.

The trigger for this article is the Northflank incident of October 30, 2026, but the steps apply to any hosting, CI or deployment platform that holds your environment variables. Everything about the incident comes from public pages we opened on November 2, 2026. We did not test the commands below against a live breach, and snippets marked "untested" are our own drafts. Run them in a non-production account first.

What do we know about the incident?

Short answer: Northflank told customers to assume all environment variables were decrypted, and users reported unauthorized use of AI keys. The root cause and the number of affected customers are not public.

The facts we could confirm, from the Northflank status page (latest entry "Monitoring", November 2, 11:09 UTC):

  • Northflank received a threat alert from Wiz on October 30 at 11:43 UTC. At 13:58 UTC it posted: "Assume all environment variables have been decrypted. Disable/rotate all keys."
  • Users reported that their AI API keys were used without authorization. BleepingComputer reported the same on October 31.
  • The company says the attack appears "focused on secrets held in our control plane", that it found no sign of access to running workloads or workload data, and that it found no sign the attacker remained. It also says it revoked access tokens, sessions, VCS integrations and cross-account IAM roles and service accounts it could act on for customers, so users must sign in again and reconnect integrations.
  • Its recommended order: AI provider keys, cloud provider credentials, payment provider keys, then private keys and database or service credentials. Revoke and replace the value at the issuing provider, then update it in the platform.

What nobody has published yet: how the attacker got in, how many customers are affected, and which secrets were used beyond the AI keys. "Assume decrypted" is a precaution, not a confirmation that your values were stolen. Treat them as compromised anyway. Rotating a secret that was never taken costs some hours of work; not rotating one that was taken can cost much more.

Wiz also published an overview of at least nine incidents at AI developer platforms in October. It mentions Northflank only in a list and gives no rotation advice, and Wiz has not confirmed that the incidents share attackers. According to BleepingComputer, two Wiz cases involved about 2.5 million Amazon Bedrock calls worth roughly $100,000 at list price. That figure comes from those cases and may not apply to Northflank customers. It does show what a stolen cloud or AI key can cost.

Where do I start?

Short answer: with a complete list of secrets, because the keys you forget to rotate are the ones that get used.

Do not rotate from memory; build an inventory first. Northflank's own guidance mentions secrets in workloads, secret groups and global secrets, so a project can hold values at several levels. Walk through every place the platform can hold a value:

  1. Per-service and per-job environment variables.
  2. Shared groups or global variables that many services inherit.
  3. Build-time variables and build secrets.
  4. Integration credentials: Git provider tokens, container registry logins, cloud account roles that the platform assumes on your behalf.
  5. Database and add-on credentials the platform created for you.
  6. Preview or staging environments, which often copy production values.
  7. Webhook signing secrets and third-party callbacks.

For each entry, record four things: what it is, who issued it (the provider that can revoke it), what it can do, and which services read it. A spreadsheet is enough. The "who issued it" column matters most, because the platform cannot revoke a key it did not issue. Deleting a variable from the platform dashboard removes your copy and revokes nothing.

If the platform has an export or API for variables, use it to avoid missing entries, and treat the output as sensitive: it contains the values you are about to retire. Do not paste it into a ticket or a chat. If you manage the platform through code, grep your repository for the variable names as a second source, for example grep -rn "process.env\|os.environ" .. That gives you names the code reads, which you can compare with the names the platform holds. This command is untested.

In what order should I rotate?

Short answer: by the damage and the speed of misuse. Anything that bills per call or moves money goes first.

Northflank's recommended order is a reasonable default, and the logic behind it holds for any platform. AI keys are billed per call, and the Bedrock figure above shows how fast a bill can grow, so misuse of the first group is likely to be both quick and expensive.

Priority Secret type Why here Check afterwards
1 AI provider keys (OpenAI, Anthropic, others) Billed per use, easy to resell or abuse Provider usage page, spend alerts
1 Cloud provider credentials (AWS access keys, service accounts) Can create resources, read data, run up large bills CloudTrail or the provider's audit log, billing
2 Payment provider keys (Stripe and similar) Money movement, customer data Key request logs, unusual refunds or charges
3 Private keys, signing keys, JWT secrets Impersonation; rotating them logs users out or breaks signature checks Token issuance, certificate logs
3 Database and service credentials Direct data access Database connection logs, new roles
4 Git, registry and CI tokens Code and supply-chain access Provider security logs
4 Webhook signing secrets, internal API keys Forged callbacks Receiver logs

Two adjustments to this order:

  • If a secret grants access to something that holds other secrets (a cloud role that can read your secrets store, a CI token that can read repository variables), move it up. Its blast radius includes everything behind it.
  • Secrets shared by many services cost more to coordinate, but do not postpone them for that reason. Schedule the deploy, do not skip the rotation.

How do I rotate without downtime?

Short answer: create the new value, deploy it, verify it, then revoke the old one. After a breach, shorten the overlap to minutes or hours and revoke at the issuer as well as in the platform.

The standard sequence works because most providers let two valid values exist for a short while:

  1. Create a new key or password at the issuer. Do this in the provider's console or API.
  2. Deploy the new value to every consumer: update the variable, then redeploy or restart the service. Environment variables are read when a process starts. The Kubernetes documentation describes automatic updates only for Secrets mounted as volumes, so a running container that received its secret as an environment variable should be assumed to keep the old value until it restarts.
    Note: If you sync secrets to your cluster using the Ennote Kubernetes Smart Agent, you can automate this. Adding an Ennote annotation to your Deployment instructs the agent to watch for secret changes and automatically trigger a zero-downtime rolling update of your pods the moment a key is rotated.
  3. Verify that traffic uses the new value. Most providers show when a key was last used. A new key with fresh usage and an old key with none is the signal you want.
  4. Revoke the old value at the issuer.

Ordinary rotation advice says to wait before deleting the old key, in case a forgotten consumer still depends on it. AWS says so in its documentation on updating access keys. It recommends that you do not immediately delete the first key and deactivate it instead, because applications still using it "will stop working" and you can reactivate it if needed. Stripe lets both the old and new key work for up to 7 days when you rotate in the dashboard.

That waiting period works against you here. The old value is the one that may have been copied. OWASP's Secrets Management Cheat Sheet says: "Keys that were exposed should undergo immediate revocation." So after a breach, the practical compromise is:

  • Prepare the deploy before you create the new key, so that the gap between creating it and using it is short.
  • Deactivate rather than delete where the provider allows it. You can reactivate within minutes if a service breaks, and the key is still dead to anyone else once it is deactivated.
  • Pick the shortest grace option. In Stripe's dashboard that is expiring the old key "Now", which deletes it, instead of the 7-day grace period. The documentation does not say this for breaches; it is our reading of the trade-off.
  • Accept that a short outage of a service you forgot to update is cheaper than a week of exposure, for services where brief downtime is tolerable. For critical ones, rehearse the swap first and update the most critical consumers first, so a missed one costs the least.

What are the steps per provider?

Short answer: each provider has its own mechanism, and the details differ.

AWS access keys. An IAM user can have at most two access keys. Create the second one, update your applications, check "Last used" with aws iam get-access-key-last-used, then deactivate the old one and delete it later:

aws iam create-access-key --user-name <user>
aws iam get-access-key-last-used --access-key-id <old-key-id>
aws iam update-access-key --user-name <user> --access-key-id <old-key-id> --status Inactive
aws iam delete-access-key --user-name <user> --access-key-id <old-key-id>

These are the commands the AWS documentation lists. Where you can, prefer short-lived role credentials over long-lived keys for the platform's integration, so there is nothing to rotate next time.

Stripe. Rotating a key in the dashboard revokes it and creates a replacement. The old key keeps working for the grace period you choose, up to 7 days, or is deleted at once if you choose "Now". Stripe's documentation also says that webhook signing secrets are not API keys and are rolled separately, in the Webhooks section, where you can expire the old secret immediately or keep it for up to 24 hours. It is easy to forget, because API key rotation does not touch it.

GitHub personal access tokens. GitHub's documentation describes deleting a token (Settings, Developer settings, Personal access tokens) and does not describe a separate revoke action. There is no regenerate action on that page, so create a new token, deploy it, then delete the old one. Deleting a token that created a deploy key also deletes that key. The security log covers the last 90 days.

Anthropic. The Admin API documentation describes API keys that you can list and deactivate by setting their status to inactive. We did not find a hard delete in the text we read. Create a new key, deploy it, and deactivate the old one. For the general order of actions for a single leaked AI key, including spend limits, see our guide on what to do when an API key leaks.

OpenAI. OpenAI's documentation was not reviewed for this article, so we do not describe its key screens here. Use the keys page in your project settings and check the usage page for the incident window.

PostgreSQL. The statement is ALTER ROLE app_user WITH PASSWORD '...'. The PostgreSQL documentation warns that the password is sent to the server in cleartext and may be logged in the client command history or the server log. The documentation says psql offers \password, which changes a password without exposing the cleartext. Changing the password and updating the app is a moment of risk: until the app has the new value, new connections fail. A safer pattern, which is our own composition and untested, is to create a second role with the same grants, point the application at it, then terminate the old role's sessions (pg_terminate_backend) and drop it. Warning: a role that owns tables cannot be dropped, and DROP OWNED BY deletes those objects, so make sure the application role does not own your tables, or use REASSIGN OWNED first. That gives you the two-value overlap that databases do not offer out of the box. If your platform created the database for you, use its managed rotation if it has one. Northflank says it enabled rotation for Postgres, MongoDB, MySQL, Redis, MinIO and RabbitMQ during the incident.

Private keys and signing secrets. There is usually no overlap built in. For JWT signing keys, check whether your verifier can accept two keys at once (for example via a key ID header). If not, rotation logs everyone out, which may be an acceptable price.

How do I check whether the keys were used?

Short answer: look at the issuer's logs and billing for the incident window, starting on October 30 in the Northflank case. A clean result is useful but not proof.

  • AWS. CloudTrail event history shows management events for the past 90 days in a region, and you can view it at no charge. Model invocation logging for Amazon Bedrock is disabled by default, so if you never turned it on, there is no invocation log to read after the fact. Use Cost Explorer and the billing console to look for unexpected model usage. We did not verify which Bedrock calls CloudTrail records, so do not rely on it alone.
  • AI providers. Open the usage and billing pages and compare against your normal traffic. Northflank's own guidance says to contact your AI providers directly if you see unexpected usage.
  • Stripe. Check the key's request log and your dashboard for charges, refunds and payouts you do not recognize. Stripe also documents access policies for keys, which can restrict by IP address and notify on unauthorized access.
  • GitHub. The security log documents personal_access_token (for fine-grained tokens) and oauth_access categories; check the page for what it records.
  • Old keys after rotation. If an old key shows use after you swapped it out, there are two explanations: a consumer you missed, or someone else. The source IP address of the requests tells them apart.

If you find misuse, move from rotation to incident response: preserve logs, cap or suspend spending, report to the provider, and decide whether customer data needs notifying under the rules that apply to you.

What goes wrong in bulk rotations?

Short answer: consumers keep the old value, the order of operations breaks something, or a secret that is not an API key is forgotten.

  • Stale processes. Services that read the variable at startup keep using the old one until restarted. Check the "last used" data on the old key and restart what you missed. If you use the Ennote Kubernetes Agent, ensure your Deployments are annotated for automatic rollouts so no stale pods are left behind.
  • Database password changed before the app. The app fails on its next connection. Where the database cannot hold two valid passwords, schedule the change with the deploy ready and accept a short failure window.
  • Cached credentials. Anything that caches a secret, such as a connection pool or a client library, needs a restart or a refresh.
  • Forgotten secrets that are not API keys. Webhook signing secrets, session secrets and internal tokens appear in the inventory only if you looked for them.
  • Cleartext in your own logs. The ALTER ROLE statement, a pasted .env file or a CI log line can leave the new secret where the old one was. Check where you typed it.
  • Rotating only inside the platform. If the platform's dashboard says "rotated" but the issuer still honors the old value, nothing changed. Confirm at the issuer.

What should change so the next breach costs less?

Short answer: hold fewer long-lived secrets in fewer places, give each one the least access it needs, and make rotation a routine you have already practiced.

The Twelve-Factor App says to store config in environment variables, and credentials for external services count as config. That convention is why platforms end up holding so many secrets. A breach of the platform then exposes all of it at once. Some ways to reduce the blast radius:

  • Scope keys. Give each service its own key with limited permissions and a spend cap, rather than one organization-wide key. A stolen key that can only call one model with a small budget is a smaller incident.
  • Prefer short-lived credentials where the provider supports them, such as assumed roles with temporary credentials instead of long-lived access keys.
  • Separate environments. Production values should not appear in preview environments.
  • Keep a single inventory. One place that records every secret, who issued it, who uses it and when it was last rotated turns the first hour of the next incident into a lookup.
  • Keep an audit trail. Northflank enabled audit logs for all customers, including free ones, on October 31. OWASP recommends logging who requested a secret, when it was used and when it was updated.
  • Practice rotation on a quiet day with one secret, so the procedure is not new during an incident.

A dedicated secret manager can hold the inventory, access rules and audit log in one place, and sync values to your workloads, including Kubernetes. It is still a service you have to trust, so apply the same questions to it: who can read values, how access is logged and how fast you can rotate.

Sources

Know what you hold before the next incident. Ennote is a secret manager for passwords, API keys, and environment variables. With role-based access, an audit log, and a native Kubernetes agent that automates zero-downtime rollouts via Deployment annotations, your next rotation becomes a one-click operation. Start for free on the Developer plan.