We had proactively set up Trivy to scan our infrastructure-as-code, looking for security vulnerabilities before they became problems. One day, the scan results came back flagged in red: unencrypted S3 buckets. High severity. The fix seemed simple - add Customer Managed Keys (CMK) with kms_master_key_id to the S3 bucket configuration and call it a day. We deployed to production, and everything seemed fine. The apps kept running. Crisis averted, right?

Not quite. Hours later, after a routine frontend deployment, four production web applications went completely dark, returning nothing but XML error pages.

But here’s the silver lining: our uptime monitoring caught it immediately. No waiting for customer reports, no delayed response. Alarms fired the instant the deployment finished. And with AI assistance, we identified the root cause and the fix within minutes—not hours of debugging through AWS documentation.

AI, AI, AI. Another story? Yes, but this one’s personal.

Sometimes the best solution to a problem creates a new problem you didn’t expect. This is a story about fixing one Terraform error, only to discover that the fix itself introduces a whole new class of deployment challenges.

It started innocently enough - a Trivy security scan flagged 9 high-severity vulnerabilities in our Terraform configuration. The issue? Unencrypted SNS topics. The fix seemed straightforward: add a KMS key, encrypt the topics, deploy to dev for validation. What could go wrong?

We’ve all been there: you start the day feeling good about wrapping up last sprint’s work, coffee in hand, ready to tackle something new. Then comes the Slack notification that changes everything.

“Hey, can you look into why a couple of Lambdas didn’t run two weeks ago?”