Facebook blamed a “faulty configuration change” for a nearly six-hour outage on Monday that prevented the company’s 3.5 billion users from accessing its social media and messaging services such as WhatsApp, Instagram and Messenger.
The company, in a blog post published late on Monday, did not specify who executed the configuration change and whether it was planned but said that its systems were back up and running.
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.”
The company said it had “no evidence” that users’ data had been compromised during the outage.
“We’re working to understand more about what happened today so we can continue to make our infrastructure more resilient,” Santosh Janardhan, the vice president of infrastructure, wrote in the post.
The Facebook outage is the largest ever tracked by web monitoring group Downdetector.
The outage was the second blow to the social media giant in as many days after a whistleblower on Sunday accused the company of repeatedly prioritising profit over clamping down on hate speech and misinformation.
As the world flocked to competing apps such as Twitter and TikTok, shares of Facebook fell 4.9 per cent, their biggest daily drop since last November, amid a broader selloff in technology stocks on Monday. Shares rose about 0.5pc in after-hours trade following the resumption of service.
“To every small and large business, family, and individual who depends on us, I’m sorry,” Facebook Chief Technology Officer Mike Schroepfer tweeted, adding that it “may take some time to get to 100pc”.
Earlier, several Facebook employees who declined to be named said that they believed that the outage was caused by an internal mistake in how internet traffic is routed to its systems. The failures of internal communication tools and other resources that depend on that same network in order to work compounded the error, the employees said.
Security experts said an inadvertent mistake or sabotage by an insider were both plausible.
“Facebook basically locked its keys in its car,” tweeted Jonathan Zittrain, director of Harvard’s Berkman Klein Centre for Internet & Society.
Other social media apps
In addition to the disruption to people, businesses and others that rely on the company’s tools, Facebook CEO Mark Zuckerberg took a financial hit.
Fortune’s billionaire tracking website late on Monday said Zuckerberg’s personal fortune plunged by nearly $6 billion from the prior day to land at just under $117bn.
For Facebook’s rivals, it was a good day, however.
The messaging service Telegram went from the 56th most downloaded free app in the US to the fifth, according to specialist firm SensorTower.
The encrypted messaging app Signal tweeted that “millions” of new users had joined, and added that it was “Signal and ready to mignal.”
Twitter on Monday reported higher-than-normal usage, which led to some issues in people accessing posts and direct messages.
In one of the day’s most popular tweets, video streaming company Netflix shared a meme from its new hit show “Squid Game” captioned “When Instagram & Facebook are down,” that showed a person labelled “Twitter” holding up a character on the verge of falling labeled “everyone.”
Inside a Facebook group for ad buyers, one member wisecracked after service returned that “lots of people searched today ‘how to run google ads for clients’.”
Facebook, which is the world’s largest seller of online ads after Google, was losing about $545,000 in United States ad revenue per hour during the outage, according to estimates from ad measurement firm Standard Media Index.
Past downtime at internet companies has had little long-term effect on their revenue growth, however.
Facebook’s services, including consumer apps such as Instagram, workplace tools it sells to businesses and internal programs, went dark at noon Eastern time (1600 GMT). Access started to return around 5:45pm ET.
Soon after the outage started, Facebook acknowledged users were having trouble accessing its apps but did not provide any specifics about the nature of the problem or say how many users were affected.
The error message on Facebook’s webpage suggested an error in the Domain Name System (DNS), which allows web addresses to take users to their destinations. A similar outage at cloud company Akamai Technologies Inc took down multiple websites in July.
On Sunday, Frances Haugen, who worked as a product manager on the civic misinformation team at Facebook, revealed that she was the whistleblower who provided documents underpinning a recent Wall Street Journal investigation and a US Senate hearing last week on Instagram’s harm to teen girls.
Haugen was due to urge the same Senate subcommittee on Tuesday to regulate the company, which she plans to liken to tobacco companies that for decades denied that smoking damaged health, according to prepared testimony seen by Reuters.
With additional input from AFP.