VS Offline? Mid-Month Outage, Explained

An explanation of the Visual Studio Online outage that occurred July 18 was simply Azure SQL Database slowness, but it was a bit more than that.

Visual Studio Online was offline for about 90 minutes on July 18, and the outage was significant enough that Microsoft Technical Fellow Brian Harry blogged an explanation of it. In short, the cause was slowness of an Azure SQL Database.

Harry explained that the outage occurred during non-peak hours so as to affect very few customers. Even so, there were some lessons to be taken away from the cause of the outage.

A simple explanation goes like this: the Visual Studio IDE was calling Shared Platform Services (SPS) "to establish a connection to get notified about updates to roaming settings"; SPS called Azure Service Bus (ASB); and ASB was calling the Azure SQL Database. The cause wasn't the services themselves, but as Harry explained, failure to "handle a transient failure in a secondary service properly and allowed it to cascade into a total service outage." In other words, one thing leads to another and problems tend to pile up rather than resolve themselves of their own accord.

Harry said that several takeaway lessons could be learned from the incident:

  • Smaller services are better: Don't pair critical and non-critical services on a shared resource; factor services into as small an atomic unit of work as possible to minimize failure points.
  • With services reliant on each other, retries will pile up exponentially. Assume the worst even if the retries seem normal.
  • Prioritize traffic and set up thresholds along the Service Bus.
  • Synchronous or asynchronous calls, threading matters.

In short: "The key thing is to examine each and every failure, trace the failure all the way to the root cause, generalize the lessons and build defenses for the future."

How do outages like the one that took down Visual Studio Online impact you or your confidence of cloud-based services? Chime in below or write me at [email protected].

About the Author

You Tell 'Em, Readers: If you've read this far, know that Michael Domingo, Visual Studio Magazine Editor in Chief, is here to serve you, dear readers, and wants to get you the information you so richly deserve. What news, content, topics, issues do you want to see covered in Visual Studio Magazine? He's listening at [email protected].

comments powered by Disqus


  • AI for GitHub Collaboration? Maybe Not So Much

    No doubt GitHub Copilot has been a boon for developers, but AI might not be the best tool for collaboration, according to developers weighing in on a recent social media post from the GitHub team.

  • Visual Studio 2022 Getting VS Code 'Command Palette' Equivalent

    As any Visual Studio Code user knows, the editor's command palette is a powerful tool for getting things done quickly, without having to navigate through menus and dialogs. Now, we learn how an equivalent is coming for Microsoft's flagship Visual Studio IDE, invoked by the same familiar Ctrl+Shift+P keyboard shortcut.

  • .NET 9 Preview 3: 'I've Been Waiting 9 Years for This API!'

    Microsoft's third preview of .NET 9 sees a lot of minor tweaks and fixes with no earth-shaking new functionality, but little things can be important to individual developers.

  • Data Anomaly Detection Using a Neural Autoencoder with C#

    Dr. James McCaffrey of Microsoft Research tackles the process of examining a set of source data to find data items that are different in some way from the majority of the source items.

  • What's New for Python, Java in Visual Studio Code

    Microsoft announced March 2024 updates to its Python and Java extensions for Visual Studio Code, the open source-based, cross-platform code editor that has repeatedly been named the No. 1 tool in major development surveys.

Subscribe on YouTube