Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 19 February 2022 09:01 Incident End: 24 February 2022 20:48 Duration: 5 days, 11 hours, 47 minutes Affected Services and Features: Google Cloud Run - .NET application failures using HTTP/2 or gRPC Regions/Zones: Global Description: Google Cloud Run services using .NET HTTP/2 or gRPC experienced elevated errors. From preliminary analysis, the root cause of the issue is a change triggered by a recent rollout. Customer Impact: Affected .Net services observed "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request was made using HTTP/2 or gRPC. Additional Details: The ability to disable "preserve_downstream_scheme" was removed upstream in Envoy proxy [1] which led to an unintentional breaking change in Cloud Run. The breaking change introduces an incompatibility with .NET default enforcement between incoming transport-layer security and “:scheme” header (see details [2] ). The Cloud Run and Envoy teams at Google were able to mitigate the issue once identified, and have taken steps to limit and better manage such changes in the future. The issue was fully resolved on 24 February 2022 at 20:48 US/Pacific once a rollback of the change was completed. [1] - https://www.envoyproxy.io/docs/envoy/latest/version_history/current#removed-config-or-runtime [2] - https://github.com/dotnet/aspnetcore/issues/30532

The issue with Cloud Run has been resolved for all affected users as of Thursday, 2022-02-24 20:57 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Thursday, 2022-02-24 US/Pacific. We will provide more information by Thursday, 2022-02-24 22:30 US/Pacific. Diagnosis: Affected customers will see "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request is made using HTTP/2 or gRPC. Workaround: If using .NET 6, settings KestrelServerOptions.AllowAlternateSchemes to true will avoid the issue: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-6.0

Summary: Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Thursday, 2022-02-24 US/Pacific. We will provide more information by Thursday, 2022-02-24 21:00 US/Pacific. Diagnosis: Affected customers will see "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request is made using HTTP/2 or gRPC. Workaround: If using .NET 6, settings KestrelServerOptions.AllowAlternateSchemes to true will avoid the issue: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-6.0

Summary: Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Thursday, 2022-02-24 US/Pacific. We will provide more information by Thursday, 2022-02-24 18:00 US/Pacific. Diagnosis: Affected customers will see "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request is made using HTTP/2 or gRPC. Workaround: If using .NET 6, settings KestrelServerOptions.AllowAlternateSchemes to true will avoid the issue: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-6.0

Summary: Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Thursday, 2022-02-24 US/Pacific. We will provide more information by Thursday, 2022-02-24 14:00 US/Pacific. Diagnosis: Affected customers will see "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request is made using HTTP/2 or gRPC. Workaround: If using .NET 6, settings KestrelServerOptions.AllowAlternateSchemes to true will avoid the issue: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-6.0

Summary: Global: Requests to .NET Cloud Run applications fail if HTTP/2 or gRPC is used Description: We are experiencing an issue with Cloud Run beginning at Thursday, 2022-02-10 18:50 US/Pacific. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We apologize to all who are affected by the disruption. We will provide an update by Wednesday, 2022-02-23 15:30 US/Pacific with current details. Diagnosis: Affected customers will see "upstream connect error or disconnect/reset before headers. reset reason: remote reset" when a request is made using HTTP/2 or gRPC. Workaround: If using .NET 6, settings KestrelServerOptions.AllowAlternateSchemes to true will avoid the issue: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes?view=aspnetcore-6.0