Config Change costs $6 billion dollars
Facebook apologized late Monday for a massive 6 hour outage that knocked out service to the social media giant’s platforms for 3.5 billion users around the world, and said the disruption was due to a “faulty configuration change.”
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Facebook said in a blog post.
And a WhatsApp employee said no internal services at company headquarters worked except for email and calendars. Even conference rooms were inaccessible during the outage, the employee said, because they’re digitally locked and unlocked through an internet-connected tablet.
The cost is tangible. Clearly there is the effort required to troubleshoot and fix the problems, and the wasted time for the staff who could work, but also there was a sell-off of Facebook shares. It sent the social-media giant’s stock plummeting 4.9% on Monday, adding to a drop of about 15% since mid-September. The stock slide on Monday sent Zuckerberg’s worth down $6 billion. He’s down to $121.6 billion from almost $140 billion in a matter of weeks, according to the Bloomberg Billionaires Index.
Including the Facebook CTO, Monday’s outage drove many social media users to Twitter. The platform responded to the influx by rolling out the digital welcome mat, saying simply, “hello literally everyone.”
But Facebook is not an isolated incident. The last major internet outage knocked many of the world’s top websites offline. The stricken content-delivery company in that case, Fastly, blamed a software bug triggered by a customer who changed a setting. Internet users were unable to access major news outlets, e-commerce platforms, and even government websites. Everyone from Amazon to the New York Times to the White House was affected, all thanks to one customer trying to change their settings.
And don’t forget the Salesforce outage which took down services including the status.salesforce.com site. They determined the root cause was related to the implementation of an emergency fix that triggered a software issue and caused a DNS network incident. To resolve the issue, the team manually restored DNS service on a data center by data center basis until normal service levels were restored.
Time To Value
Config Management is more than just tracking changes or preventing outages. Taking a broader perspective, it means that every change should be taking the business forward – as quickly as possible. That is what Time To Value means. It starts with business analysis to make sure the right changes are being made. Those changes that the business needs, not the ones that they thought they wanted. It then requires org impact analysis to assess the risk of making a change from a business, technical and regulatory basis. This means that the user stories that are passed to DevOps are accurate and risk-assessed. It superpowers DevOps.
We call that a Change Intelligence Platform. Elements.cloud is #1 Change Intelligence Platform for Salesforce but also for other enterprise applications e.g. Pardot, Slack, Jira, ServiceNow or Workday.
You don’t need a $6 billion business case
And the great news is it is affordable for even the smallest Salesforce implementation. Talk to us and find out how to accelerate your future.