In-Depth
MSDN Redux With 64-Bit Servers and ASP.NET 2.0
Microsoft's Larry Jordan describes how his team reinvigorated MSDN with 64-bit servers, VS 2005, and Web services.
Larry Jordan, Microsoft's Development Manager for the MDSN and TechNet Development Team, billed MSDN as "the source of truth in technical documentation" during his Wednesday keynote presentation at VSLive! 2006 San Francisco. Prior to the complete MSDN site makeover by Larry and his team that began two years ago, relying on MSDN Search to seek the "truth" frustrated developers to no end. According to a show of hands from the keynote audience, most developers use Google to find the MSDN software documentation pages they need.
MSDN Search is just one of the site's technologies to be replaced; you can test the beta version of the new MSDN search engine at the MSDN Lab site (see Figure 1 and Resources). The MSDN team ported the original ASP site to ASP.NET 2.0, migrated document storage from the Windows file system to 32-bit SQL Sever 2005, and wrote XML Web services to serve as an API for metadata and document access.
The MSDN group consists of about 180 people who are responsible the aggregation of all Microsoft technical documentation for online and CD/DVD subscriptions. According to Jordan, "MSDN receives technical documentation in nine different languages from several hundred sources within Microsoft. Incoming documentation is in many disparate formats across several delivery mechanisms. MSDN has had a static content acquisition and publishing process for the last five years." Jordan admitted that "it's embarrassing that MSDN has been on ASP for four and a half years. ... We got ourselves into a brittle space. The documentation and bat files meant that we could not easily pull ourselves out of that."
According to Jordan, ASP.NET 2.0 and SQL Server 2005 have enabled his 28-developer team to:
- Deliver tens of millions of unique topics from a data-driven site
- Offer new ways to access relevant technical documentation, such as F1 from VS 2005
- Introduce an improved feedback loop to measure user satisfaction with documentation refresh activities
MSDN currently serves about 75 million page views to 8 million different users per month from 3 million pages. Traffic levels are consistent during weekdays, but spikes occur in the morning around the world.
Using ASP.NET 2.0 Controls Out-of-the-Box
Jordan says that MSDN's original ASP version primarily aggregated XML documents and rendered Web pages, but ASP lacked templating and other features that were needed for a production Web site that runs a central set of services. The MSDN team began migrating the site with ASP.NET 2.0 pre-beta 1 bits and completed the task by eliminating proprietary code for TOC management, site maps, and rendering. The current MSDN site is "100 percent out-of the box ADO.NET 2.0 components." Removing proprietary code lets other developers replicate many of the MSDN team's content-management techniques.
Andy Oakley, lead program manager for the MSDN rewrite, described these programming components implemented or enabled with managed .NET 2.0 code:
- Rendering
- Virtual path provider, a critical part of the infrastructure which enables database storage
- Master pages to support CSS and provide a consistent look and feel
Programmatic API used with SQL Server 2005 and VS 2005 to deliver a seamless client experience
- Online help integration
- Content discovery
- Content delivery
Content acquisition that assures XHTML compliance to enable transforms for a consistent look and feel
Database storage and connectivity, which enables inventorying content
Aggregation system with offline data processing
Oakley reckoned that the ASP version caused "the 404 page [to be] one of the most popular pages on MSDN" and moving to ASP.NET 2.0 has solved this problem. The current content is expected to have a lifetime of 10 to 15 years, which minimizes "not found" occurrences. Oakley described how MSDN has replaced unfriendly GUIDs in document URLs with more easily typed eight-digit unique identifiers. Oakley also said that you can use custom URL aliases, such as http://msdn2.microsoft.com/system.string, to open the help topic for any .NET 2.0 namespace. The /namespace/object syntax only works for the new MSDN2 cluster (see Resources). He also mentioned that you can add @MSDN, http://msdn2.microsoft.com/$w as an MSN desktop search shortcut in the deskbar search box and then just type msdn system.string in the deskbar search box to open the topic.
Drilling down on the rendering and delivery topics, Oakley cited these-ASP.NET 2.0 advantages:
- High-performance rendering
- Smaller data center footprint
- Adaptive caching for less frequently used pages
- Master pages for varied delivery
- 'Chromeless' rendering in the application's UI, such as VS 2005's F1 online help
- Printer-friendly output
- Optimized for search engines
- Firefox friendliness
Rating and feedback mechanism available on every page
Oakley said that the current system provided a "several-fold" performance increase. MSDN is now delivering 130 requests per second to 6,000 concurrent users with a mean rendering time 0.4 seconds (worst case) with no page caching—in this case, each request fetches the page from the database server and renders it to each user.
Moving to 64-bit hardware for the operating system, rendering, and database was a major contributor to the improved performance and reduced the number of servers required. The data center topology employs 16 Web servers and four SQL Servers for redundancy, but two Web servers and a single database server can handle the current load. In normal operation, the output rendering cache increases to about 10 GB RAM after a day of operation, so the cache delivers most pages much more quickly than 0.4 seconds. A single staging server database uses an indexed view and replication to update 300,000 pages on all database servers in about 45 minutes.
Taking Advantage of Web Service and Database Features
Jordan expanded on his explanation of the forthcoming MSDN Web services that he originally described at VSLive! 2005 Orlando and that came on line (privately) in late January. When the Web services become publicly available later this year, they'll offer these features:
- Web service interface supports
- Add VS 2005 Web Reference
- A variety of cross-platform SOAP toolkits
- RESTful Ajax interfaces
Complete services for
- Content discovery
- Search
- Metadata
- Features
- Images
- Ratings
- Aggregation
Jordan then demonstrated a simple Windows form Web service client to consume the MSDN services. Keep an eye on the MSDN Lab home page for MSDN Web services availability (see Resources). Another group is in the process of creating a developer preview of a rich WinFX MSDN Web service client that takes full advantage of Windows Presentation Services.
Finally, Jordan described the new SQL Server 2005 features that contributed to the MSDN makeover:
- READFAST locking hints to skip records that are locked, which is great for rolling your own queue
- OUTPUT clause that writes values resulting from INSERTR, UPDATE or DELETE statements to a specified table variable or temporary table
- Persisted computed columns that permit compound columns to reside in the data page
- Include keyword on CREATE INDEX statement
- Common Table Expressions (CTEs) that enable recursive queries or simplify redundant table joins
The Jordan and Oakley test-case presentation might have been improved by providing before-after-ASP.NET 2.0 comparison metrics and detailed specifications for the datacenter hardware and its configuration. Regardless of this omission, the two team leaders made a strong case for upgrading to the 64-bit versions of Windows Server 2003 R2 Enterprise Edition, Visual Studio 2005, and SQL Server 2005 Enterprise Edition for large-scale, Web-based content management systems.