Visual Studio Team Services Stops Twice

When your goal is 99.999-percent uptime, there's a small crack in the window for downtime. That happened with Visual Studio Team Services last week. And it happened within a span of two days.

When Microsoft's cloud services guarantee is for 99.999-percent uptime, it means the window is cracked open ever so slightly for a period of downtime. That happened with Visual Studio Team Services withiin a span of two days just recently. Microsoft's Brian Harry detailed both stoppages in a pair of detailed blog posts (here and here).

The first stoppage occurred on February 3, at 3:30 GMT, with worldwide impact to the service. This incident kept VSTS quiet for three hours. As Harry noted in a blog, "We saw a large spike in response times from less than 10% of the requests coming from the browser. At this point alerts fired and we engaged our on-call DRIs." He said that the issue centered on missing Azure DNS entries, which occurred because of a change -- which, unknowingly, introduced a bug -- in how the team performs automated cleanup of DNS entries for inactive accounts. And this problem cascaded to another separate but related incident which was "was mitigated by failing over to the secondary SQL server."

Then on Feb. 4, at 9:11 GMT, users in large numbers started reporting login issues and slowness of response from VSTS. "The root cause is that we (the Team Services team) changed the SQL Azure query processor compat level from 100 (SQL Server 2008) to 120 (SQL Server 2014) on one of the SPS databases," said Harry. The change cascaded to other problems, which started to eat up memory as queries started to come in from users. From there, he explains in a fairly detailed and visual manner the incident response timeline up to final mitigation, which spanned five hours.

What is interesting about the two incidents is the thoroughness of reporting that Microsoft provided on these stoppages. "Those two post are a fascinating insight into incident response, problem solving and lessons learned," notes one commenter named Cedric. " They make a very interesting read for anybody involved in systems architecture and incident response team.

One interesting tidbit from the incidents that Harry revealed is that the visualizations he used to illustrate the problems he described in the blogs were created by a tool that's part of Application Insights called "Kusto," at least while it's in development. Harry explains: "Pay attention at the //Build/ conference.  We're going to be talking about it.  We've already got many dozens, maybe hundreds of services across Microsoft using it and it is ingesting/querying ~300TB of telemetry per day and growing VERY rapidly."

About the Author

You Tell 'Em, Readers: If you've read this far, know that Michael Domingo, Visual Studio Magazine Editor in Chief, is here to serve you, dear readers, and wants to get you the information you so richly deserve. What news, content, topics, issues do you want to see covered in Visual Studio Magazine? He's listening at [email protected].

comments powered by Disqus


  • AI for GitHub Collaboration? Maybe Not So Much

    No doubt GitHub Copilot has been a boon for developers, but AI might not be the best tool for collaboration, according to developers weighing in on a recent social media post from the GitHub team.

  • Visual Studio 2022 Getting VS Code 'Command Palette' Equivalent

    As any Visual Studio Code user knows, the editor's command palette is a powerful tool for getting things done quickly, without having to navigate through menus and dialogs. Now, we learn how an equivalent is coming for Microsoft's flagship Visual Studio IDE, invoked by the same familiar Ctrl+Shift+P keyboard shortcut.

  • .NET 9 Preview 3: 'I've Been Waiting 9 Years for This API!'

    Microsoft's third preview of .NET 9 sees a lot of minor tweaks and fixes with no earth-shaking new functionality, but little things can be important to individual developers.

  • Data Anomaly Detection Using a Neural Autoencoder with C#

    Dr. James McCaffrey of Microsoft Research tackles the process of examining a set of source data to find data items that are different in some way from the majority of the source items.

  • What's New for Python, Java in Visual Studio Code

    Microsoft announced March 2024 updates to its Python and Java extensions for Visual Studio Code, the open source-based, cross-platform code editor that has repeatedly been named the No. 1 tool in major development surveys.

Subscribe on YouTube