Developer's Toolkit

Blog archive

Architecting for Scalability

I had heard Pat Helland speak several times over the last couple of years, when he served Microsoft, and enjoyed his Megalopolis talk. That presentation compared an enterprise architecture to the evolving infrastructure of a city, putting the architect in the role of city planner and government official.

Well, Pat moved on to Amazon last year, but his talk this week on Architecting for Scalability was as entertaining and thought-provoking as any I had heard from his Microsoft days. One thought in particular that sticks in my mind was his assertion that it might be time to rethink some of the accepted wisdoms concerning data management.

For example, he notes that data normalization is not necessarily a good thing. Normalization, which manifests itself (in one way) as the requirement that data should only be changed in one location in the database, makes sense only if you're planning on changing that data, or, as Pat put it, executing UPDATE WHERE . . .

In many cases, that is a normal database activity. But he points out that is changing. First, offline storage (especially volatile storage) is cheap, enabling people to more easily save whole copies of databases for archive. Second, more businesses have to save database information as it is, rather than continually update it. Humorously, he notes that not only are business required not to update certain databases, but in the Sarbanes-Oxley era, doing so can be a felony (not so humorous, I suppose, if you are the one declared a felon).

But the point is that data management requirements are changing dramatically, and both architects and DBAs have to understand the implication to their jobs. In the case of architects, it has implications to how you go about architecting an application. Scalability has a different meaning when you have to treat data differently. In Pat's case, he notes that it becomes more important to not apply data across different business entities. It doesn't necessarily matter what database that data comes from; rather, it depends greatly on how it is used.

I don't normally get excited about data management, but I am in this case for two reasons. First, I work for a company (Progress) that produces a non-normalized database (although it can be validated for most of the behavior of the third normal form), so it was interesting to hear conventional wisdom being questioned in that regard. Second, it is always interesting to watch business requirements change and observe how both technology and skills adapt. We're still in the early stages here, so keep your eyes open to see how database technologies and best practices shift over the coming years to adapt to new business and legal standards.

Posted by Peter Varhol on 02/01/2006 at 1:15 PM


comments powered by Disqus

Featured

  • Move Over, Stack Overflow: Microsoft Launches Q&A for .NET

    Stack Overflow probably isn't worried, but Microsoft has launched its own Q&A site for all things .NET, seeking to provide a one-stop-shop for getting .NET technical questions answered by the community.

  • Developer Decries WinForms-to-Blazor Performance Degradation

    Since shipping .NET 5, Visual Studio 2019 v16.8 and more goodies recently, Microsoft has been touting speed improvements in many components -- including the red-hot Blazor project -- but some real-world developers are finding different results.

  • Google Cloud Functions Supports .NET Core 3.1 (but not .NET 5)

    Google Cloud Functions -- often used for serverless, event-driven projects -- now supports .NET, but the new support is a release behind Microsoft's latest .NET offering.

  • Binary Classification Using PyTorch: Model Accuracy

    In the final article of a four-part series on binary classification using PyTorch, Dr. James McCaffrey of Microsoft Research shows how to evaluate the accuracy of a trained model, save a model to file, and use a model to make predictions.

  • Visual Basic in .NET 5: Ready for WinForms Apps

    With the milestone .NET 5 and Visual Studio 2019 v16.8 releases now out, Microsoft is reminding Visual Basic coders that their favorite programming language enjoys full support and the troublesome Windows Forms Designer is even complete -- almost.

Upcoming Events