A massive power outage yesterday saw a UKFast outage at their MANOC 5 Datacentre. The ongoing issues over the next 24 hours have seen many businesses lose a whole day’s trading in the busiest time of the year. It appears that workmen outside of the UKFast facilities might have wielded a pick axe too enthusiastically and accidentally cut through two power supply cables.
UKFast told customers “Incoming mains supply has been lost to the site and our engineers are currently investigating why the generators have not taken over service” and then at 11am added “Power is currently restored via Generator supply. Our engineers are now powering on solutions”. One would expect that Uninterruptible Power Supplies (basically batteries) should prevent loss of power whilst the back up generators kicked in but this didn’t happen. It’s not yet known why the back up generators failed to start up quickly enough and take the load and the delay appears to have caused a ton of knock on issues which are still being resolved.
Throughout the day updates started with “Engineers are continuing to resolve residual issues…” as different services were bought back on line. As of now the final update from 3.15am today reads “Our engineers are continuing to bring back the remaining services”.
The effect of the UKFast outage on OnBuy
UK Marketplace OnBuy who invested heavily in infrastructure and relied on UKFast to host their services were off line for 11 hours. You’d think a British marketplace would invest in British companies and they did, but it didn’t go to plan this time.
The power came back in an hour – which was terrible, and should never have happened, but that’s not what caused the biggest problem. An hour was damaging and would need serious explanation, but it actually took us a further 10.5 hours to get trading. We were in our office until gone midnight, waiting for UKFast to get OnBuy back online. It was very painful and frustrating.
– Cas Paton, OnBuy
OnBuy are just one of the thousands of customers who trust UKFast to run their websites – many users will be much smaller and bizarrely may have got back up and running more quickly as UKFast’s eCloud Hybrid service which OnBuy use was one of the last to come back online.
Were you affected by the UKFast Outage? Perhaps not your own website, but if you were greeted by a 503 error from a UK website yesterday it could well have been one hosted by UKFast.
Amazon might have paid of those “power supply” workers to cut a few cables. Sucks when you have just launched a marketing campaign.
My partners whole travel agency was down yesterday also nationwide, she does say they outsource also but has know idea who.
And deactivated the backup generators?
We were down for 3:30hrs
Should use a proper companies like Microsoft azure or Amazon aws.
Yet another reason to stay clear of OnBuy. They can’t even pick a good company to host it’s systems.
This isn’t the first time UKfast have experienced problems, we have made the decision to move to a new smaller independent site owned by itility based in Manchester.
Couldn’t risk it anymore with these guys and loved the flexible and welcoming approach of itility who made the whole process extremely easy.
We went down for the whole day!
Luckily marketplace sales kept us afloat 😉
Probably be moving over to Amazon Web Services now…
Comments are closed.