News
Visual Studio Team Services Stops Twice
When your goal is 99.999-percent uptime, there's a small crack in the window for downtime. That happened with Visual Studio Team Services last week. And it happened within a span of two days.
When Microsoft's cloud services guarantee is for 99.999-percent uptime, it means the window is cracked open ever so slightly for a period of downtime. That happened with Visual Studio Team Services withiin a span of two days just recently. Microsoft's Brian Harry detailed both stoppages in a pair of detailed blog posts (here and here).
The first stoppage occurred on February 3, at 3:30 GMT, with worldwide impact to the service. This incident kept VSTS quiet for three hours. As Harry noted in a blog, "We saw a large spike in response times from less than 10% of the requests coming from the browser. At this point alerts fired and we engaged our on-call DRIs." He said that the issue centered on missing Azure DNS entries, which occurred because of a change -- which, unknowingly, introduced a bug -- in how the team performs automated cleanup of DNS entries for inactive accounts. And this problem cascaded to another separate but related incident which was "was mitigated by failing over to the secondary SQL server."
Then on Feb. 4, at 9:11 GMT, users in large numbers started reporting login issues and slowness of response from VSTS. "The root cause is that we (the Team Services team) changed the SQL Azure query processor compat level from 100 (SQL Server 2008) to 120 (SQL Server 2014) on one of the SPS databases," said Harry. The change cascaded to other problems, which started to eat up memory as queries started to come in from users. From there, he explains in a fairly detailed and visual manner the incident response timeline up to final mitigation, which spanned five hours.
What is interesting about the two incidents is the thoroughness of reporting that Microsoft provided on these stoppages. "Those two post are a fascinating insight into incident response, problem solving and lessons learned," notes one commenter named Cedric. "
They make a very interesting read for anybody involved in systems architecture and incident response team.
One interesting tidbit from the incidents that Harry revealed is that the visualizations he used to illustrate the problems he described in the blogs were created by a tool that's part of Application Insights called "Kusto," at least while it's in development. Harry explains: "Pay attention at the //Build/ conference. We're going to be talking about it. We've already got many dozens, maybe hundreds of services across Microsoft using it and it is ingesting/querying ~300TB of telemetry per day and growing VERY rapidly."
About the Author
You Tell 'Em, Readers: If you've read this far, know that Michael Domingo, Visual Studio Magazine Editor in Chief, is here to serve you, dear readers, and wants to get you the information you so richly deserve. What news, content, topics, issues do you want to see covered in Visual Studio Magazine? He's listening at [email protected].