Google Cloud hit by errors and network failures.
Newsflash: Google’s cloud computing services have been hit by technical problems in Europe.
Google reports that “multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region”, and it does not currently have a work around.
The europe-west2 region is London, according to Google’s Service table.
It says:
Incident affecting BigQuery Data Transfer Service, Cloud Filestore, Cloud Run, Google App Engine, Google Cloud Dataflow, Google Cloud NetApp Volumes, Google Cloud Pub/Sub, Google Cloud Storage, Google Compute Engine, Google Kubernetes Engine, Looker (Google Cloud core), Managed Airflow, Managed Service for Apache Kafka, Persistent Disk, Storage Transfer Service
The problem is hurting some fintech companies, and banks.
Key events
Google: power event disrupted our Cloud services
Google are blaming a ‘power event’ for the disruption at its London-based Cloud computing operations today.
In an update on its status page, Google says:
Due to a power event we had suffered an outage within europe-west2-b zone. The power event is mitigated. Partial mitigation to infrastructure services will be observed within the next 30 minutes and the remaining services will be mitigated there after.
PA: Court hearings delayed as nationwide video system goes down
The video system used by courts across England and Wales went down on Tuesday in an issue believed to be linked to the wider Google outage, the Press Association report.
The Cloud Video Platform (CVP), which is used by defendants, lawyers and other parties to attend court hearings remotely, was not functioning throughout the morning.
Users expecting to attend court on video link were met with a message saying there was an internal server error.
The issue is expected to have left many hearings across the country delayed or unable to go ahead as the system is often used for defendants who are in custody to attend court.
The judge in the Russell Brand trial, Mr Justice Bennathan, decided to press on with proceedings at Southwark Crown Court on Tuesday morning despite the link being broken.
Some journalists who would have been watching online were squeezed into extra seats in court, but many were not able to follow the proceedings as the fourth complainant started her evidence.
Staff at the court said there was a national outage and did not know when it would be resolved.
It is understood to be linked to an outage affecting Google’s Cloud network, which affected businesses including online bank Revolut, delivery service DPD and Royal Mail.
The nature of cloud ecosystems mean that a problem in one region can ripple out and cause widesread disruption.
Dr Amro Al-Said Ahmad, lecturer in computer science at Keele University, explains:
“Another outage, another reminder of the fragility of cloud infrastructure. Google Cloud is currently experiencing a major incident affecting the europe-west2 (London) region, with elevated errors and intermittent network failures impacting more than 38 cloud-native services across key categories including Compute & Containers, Storage & Databases, Analytics & Data Processing, Networking & Connectivity, Messaging & Streaming, Caching & Memory Services, and AI/Platform Services.
“Google Engineers are actively investigating the issue and testing mitigations, including moving impacted workloads to alternative regions where feasible. The disruption has also affected several major organisations and financial institutions that rely on the affected region.
“More importantly, the incident highlights the dependency effect of modern cloud ecosystems. When a core cloud region experiences failures, the impact extends far beyond the directly affected services. Applications, APIs, databases, authentication systems, payment platforms, analytics pipelines, and third-party SaaS products that depend on those services can all experience cascading failures.
“A problem in a single cloud region can therefore propagate through an entire digital supply chain, affecting organisations that are several layers removed from the original outage. While this is a regional outage centred on europe-west2, the breadth of affected services highlights a key cloud engineering lesson: resilience is not achieved by simply moving to the cloud. True resilience requires designing for regional failures through multi-region deployments, workload portability, and tested disaster recovery procedures.”
Julia Kollewe
A Lloyds spokesperson has confirmed the bank (which includes Halifax) has experienced problems today:
“Our app is online and customers can log in normally, though we understand some are having issues with certain online services this morning. We are aware of similar issues affecting other organisations and are monitoring the situation closely.”
Google’s latest update on its Cloud outage reiterates what we knew already.
It says:
We apologize to all who are affected by the disruption.
Symptoms
Multiple Google Cloud services are experiencing elevated errors and intermittent network failures in europe-west2 region.
Note: Looker Studio is not impacted by this outage.
Workaround
If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).
ESET: cloud services are a ‘huge point of failure’
Today’s Google Cloud outage comes exactly 12 months after rival Amazon suffered serious disruption at its cloud computing arm.
Now, as then, it’s a reminder of the perils of relying on a small number of companies for operating the global internet, especially as more and more services have been moved onto the cloud.
Jake Moore, Global Cybersecurity Adviser at ESET, Europe’s leading cybersecurity company:
“Today’s outages highlight just how much dependence we put on a handful of technology giants to keep our digital lives running. With only a small number of major cloud providers to choose from, the knock-on effects can be catastrophic if something goes wrong and can leave millions of people unable to use everyday services.
“Cloud services are a huge point of failure and will always be a potential weakness in digital infrastructures. They could even become a potential target on such services by cybercriminals. Businesses therefore need to plan for when these services go down and have strict contingencies in place for this unfortunate inevitable event.”
We’re expecting another update from Google by noon UK time (4am PST).
The Register: Royal Mail, DPD, BT, Cloudflare and Screwfix also affected
Other companies affected by the Google Cloud outage include Royal Mail, DPD, BT, Cloudflare and Screwfix (on top of the companies we’ve already covered), according to tech news site The Register.
El Reg (as they’re affectionately known) adds:
Techies on Reddit said the interruption was having a cascading affect on multiple suppliers. One said: “oh man the classic one zone takes a nap and your non-HA service is in it.”
“Loads of stuff seems to be down, DNS somewhere,” mused another.
Some Halifax customers are finding they cannot log into their banking appp right now – and receive this error message instead:
Halifax: some customers are having issues with our app and online
Halifax, which is part of Lloyds Banking Group, has confirmed some customers are having problems today.
Posting on X, Halifax says:
We’re aware some customers are having issues with our app and online. We’re sorry about this and are working hard to fix it. If you need to you can still call us. We’ll let you know as soon as it’s back to normal.
There are also reports that Rightmove has been affected by the Google Cloud outage.
Rightmove certainly uses Google Cloud for its new artificial intelligence services – including a new AI-powered Comparison Assistant launched earlier this month.
It has also been developing a AI-powered conversational property search “in close collaboration with Google Cloud”.
NewsThump, the satire and spoof news website, has been thumped by the Google Cloud outage – its site currently won’t load.
It says:
NewsThump has been down since 7am due to a major outage in the Google Cloud infrastructure (no, we don’t really understand that either). We have satirists chomping at the bit, and jokes to post as soon as everything comes back online…
Google suggests a workaround
Google has updated its advice to those affected by its service disruption, with a possible workaround.
It says:
If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).
“A person familiar with the matter” has told Reuters that Revolut’s service outage was related to the Google Cloud service disruption (as we speculated a little while ago…)
There are also some reports of problems with other UK financial services firms today – perhaps a sign that the Google Cloud outage is causing wider problems.
These chart from DownDetector show a rise in problems being reported by users of HSBC and Lloyds Banking Group this morning:
