Increased Lambda invoke error rates

We can confirm increased invoke errors in the EU-CENTRAL-2 Region. We have identified the root cause and are actively working to resolve the issue. We will provide an update within the next 45 minutes or as new information becomes available.

We are observing significant recovery of invoke errors in the EU-CENTRAL-2 Region. Lambda functions deployed using ZIP files are now operating normally. Functions using OCI-based images continue to experience invoke failures as well as Create and Update function failures. For business critical functionality, customers can fall back to using ZIP based Lambda functions. We are actively deploying targeted fixes to restore functionality for OCI-based Lambda functions. We will provide an update within 45 minutes or as new information becomes available.

We continue to see significant recovery of CreateFunction, UpdateFunction API calls and invokes for ZIP-based Lambda functions. During our mitigation of the remaining impact to OCI-based Lambda functions, we have identified an additional symptom that may also impact some ZIP-based Lambda functions when they are being updated. While we're working to resolve remaining impact to ZIP-based Lambda functions, customers who need immediate recovery can delete and recreate these functions to unblock this impact. While impact remains to OCI-based Lambda functions, we are actively deploying a fix aimed to fully recover this. We will provide an update within 45 minutes or as new information becomes available.

We continue to experience increased error rates for CreateFunction, UpdateFunction, and Lambda invocations affecting OCI-based Lambda functions primarily and UpdateFunction errors for a subset of ZIP-based Lambda functions in the EU-CENTRAL-2 Region. We have identified a fix and are conducting thorough testing before deployment to prevent any service degradation. We expect to begin deploying our fix at or before 7:15 AM PDT, and customers will see gradual improvements as the deployment progresses. For customers who need immediate recovery for ZIP-based function updates, deleting and recreating these functions will unblock this impact. We will provide an update within 45 minutes or as new information becomes available.

Starting at 2:30 AM PDT, we began investigating increased error rates for CreateFunction, UpdateFunction, and Lambda invocations affecting OCI-based Lambda functions primarily and UpdateFunction errors for a subset of ZIP-based Lambda functions in the EU-CENTRAL-2 Region. As of 7:00 AM PDT we have begun deploying a fix aimed at resolving this issue and are monitoring the deployment closely. Customers will see gradual recovery as the fix deploys. We will provide an update within 45 minutes or as new information becomes available.

We continue to monitor our progress towards full recovery. We're tracking the final changes based on our fix and should have a more definitive update for customers in the next 45 minutes.

Between 2:34 AM and 8:47 AM PDT, we experienced increased error rates for CreateFunction, UpdateFunction, and Lambda invocations in the EU-CENTRAL-2 Region. Engineers were immediately engaged and began working to identify on multiple parallel paths to root cause and mitigate the issue. Initially the impact was to all new Lambda invocations, but we mitigated ZIP-based function impacts at 4:23 AM PDT. After 4:23 AM PDT, we continued to mitigate impact to OCI-based Lambda functions, as well as to recover a subset of ZIP-based Lambda functions that also experienced UpdateFunction errors. During this time, customers may have received an Insufficient Capacity Error. By 7:54 AM PDT, we began the deployment to fully mitigate the issue, and had fully mitigated the issue by 8:47 AM PDT. We recommend customers retry any failed operations. The issue has been resolved and all services are operating normally.