Developers Learn What's New in the MongoDB 2.6 NoSQL Database Release

Developers today learned more about the recently released MongoDB version 2.6 when parent company MongoDB Inc. conducted a webinar to explain its new features.

Last week the company announced the "biggest release ever" of its popular NoSQL database, with improvements coming in the areas of aggregation, text-search integration, query engine performance, security and many more.

"With comprehensive core server enhancements, a groundbreaking new automation tool, and critical enterprise features, MongoDB 2.6 is by far our biggest release ever," said Eliot Horowitz, CTO and co-founder.

With standard and enterprise editions, MongoDB is an open source, document-oriented database popular in Big Data implementations. Instead of storing data in tables or columns, the alternative to a traditional relational database management system (RDBMS) uses JSON-like documents with dynamic schemas, what the company calls BSON, short for binary JSON. The company claims MongoDB is the leading NoSQL database, based on metrics such as user searches, job postings and comparison of the skills listed by LinkedIn members, among others.

In today's webinar, company vice president Dan Pasette joined Horowitz to explain the new features of special import to developers. Pasette said most of the 500-plus enhancements to version 2.6 -- which was in development for about a year -- wouldn't be that noticeable to coders, aside from increased performance.

Pasette, who said the MongoDB engineering team numbered about 15 or 20 when he first started at the company but now has more than 100, cited aggregation pipeline enhancement as "one of the most compelling features." It's been around for two versions, he said, but developers can now use it to unlock larger result sets. These had been limited to 16MB, Pasette said, but developers can now get results back with a cursor, which means they can be of unlimited size. Also, he said, developers can use the $out feature to pipe results into a new collection.

Other improvements developers would be interested in, Pasette said, were index aggregation and integrated text search, which is now fully available and directly integrated in queries and pipelines. Also, he said, developers can now use new update operators, such as $multiple and $min.

Horowitz said the company -- originally called 10gen -- used its five years of experience garnered from hundreds of thousands of deployments to create the groundwork for more innovation in the next 10 years.

"You'll see the benefits in better performance and new innovations," Horowitz said. "We rewrote the entire query execution engine to improve scalability, and took our first step in building a sophisticated query planner by introducing index intersection. We've made the codebase easier to maintain, and made it easier to implement new features. Finally, MongoDB 2.6 lays the foundation for massive improvements to concurrency in MongoDB 2.8, including document-level locking."

According to release notes, both editions of MongoDB feature support for variables and new operations that will handle sets and redact data.

Text search is enabled by default in the new release, while the query system now includes a $text operator to resolve text-search queries.

Insert and update improvements include new operations and better consistency of modified data.

A new write operation protocol features improved support for bulk operations and also integrates write operations and write concerns.

Security improvements include better SSL support, x.509 authentication, an enhanced authorization system that features more granular controls, centralized storage of credentials and better tools for user management. The new version also features TLS encryption, along with user-defined roles, auditing functionality and field-level redaction, which Horowitz described as "a critical building block for trusted systems." The database auditing feature is extended by the new capability to integrate with IBM InfoSphere Guardium.

The improved query engine features a new index intersection that will fulfill queries that are supported by more than one index. Also, index filters will limit the indexes that can "become the winning plan for a query." Developers using the database can now use the count method in conjunction with the hint method. You can learn more about that here.

Also counted in the raft of improvements are better geospatial support, index build enhancements, and augmented sharding and replication administration.

Ted Neward, an expert database developer, author and presenter, told me he thought "most of the features introduced in this release are more about Mongo 'growing up' as an enterprise-class database, and it's less developer-centric than previous releases. Consider: a new security model, several administration features, and LDAP support and SNMP functionality. These are the things that a world-class enterprise has come to expect of their data storage systems, and this removes significant blockers for many companies."

Horowitz also emphasized the benefits of the new release to IT operations staffers.

"From the very beginning, MongoDB has offered developers a simple and elegant way to manage their data," Horowitz said. "Now we're bringing that same simplicity and elegance to managing MongoDB. MongoDB Management Service (MMS), which already provides 35,000 MongoDB customers with monitoring and alerting, now provides backup and point-in-time restore functionality, in the cloud and on-premises."

Neward said the emphasis on operations is good "because it signals that MongoDB Inc. is being persuaded/pressured/forced (depending on whose view you take) to put the necessary features in that commercial-scale enterprises need, which in turn signals a growing culture of adoption around non-relational data stores."

Horowitz last week also teased an automation feature for hosted and on-premises options, coming later this year. He said the "game-changing" automation functionality "will allow you to provision and manage MongoDB replica sets and sharded clusters via a simple yet sophisticated interface."

Is the new MongoDB 2.6 now enterprise-ready? Comment here or drop me a line


Posted by David Ramel on 04/17/2014 at 1:41 PM0 comments

Devs Blast Microsoft Turnaround on WCF Data Services and OData

Developers today were still weighing in on a recent Microsoft decision to switch directions on OData support to focus on ASP.NET Web API instead of WCF Data Services, angering many who had invested much work into the latter.

OK, this is somewhat involved, but I think it deserves a close look because it touches on so many issues developers have with Microsoft's support of new products and technologies and changes in direction. In this case, data devs in particular.

In a nutshell, Microsoft last week announced it will shift its OData-related development tool efforts from WCF Data Services to ASP.NET Web API, moving WCF Data Services to open source for further development. That angered many frontline developers whose teams or companies have invested a lot of work in WCF Data Services and now feel abandoned. They now have to consider their options and move forward with a new strategy. Sound familiar?

First, to get everybody on the same page regarding terminology, including those readers who might not be data developers, I'll provide brief descriptions of the terms.

WCF Data Services lets data devs easily build services that use the open OData protocol to expose and consume data over the Web using REST semantics.

REST is an architectural style relying on a stateless, cacheable communications protocol, basically HTTP. Instead of using SQL commands, for example, you can get a list of customers from the sample OData Northwind database service's Customers table by sending the URL:

OData is an open, standardized protocol for providing and consuming data APIs using HTTP and REST.

ASP.NET Web API is described by Microsoft as an ideal platform for creating RESTful applications on the Microsoft .NET Framework.

OData recently advanced to version 4.0. Microsoft began work on WCF Data Services to support the new features. It ran into two major problems, as explained in the announcement blog by Michael Pizzo, principal architect, Microsoft OData Team:

First, it required that the underlying data source support fairly rich query semantics. This made it difficult to implement over diverse sources of data such as data returned from a function invocation or collected from other services. Even for queryable sources, computing the best query to perform on the underlying store was often difficult because the query passed to the data provider lacked a lot of the context of the initial request.

Second, because WCF Data Services was somewhat of a monolithic "black box," services had no way to expose new features until supported for each feature was added to the WCF Data Services stack. Adding support to WCF Data Services often in turn required defining new interfaces that each data provider would then have to implement before the functionality was available to the service.

Long story short: Microsoft is putting its OData weight behind ASP.NET Web API as the tool of choice for creating OData services. Using the model-view-controller (MVC) pattern, service developers have much more control over data requests by implementing custom controllers for each route. This will require developers to write more code, Pizzo admitted, but much of that is boilerplate code and developers can get a head start writing it via controller scaffolding.

"We didn't make this decision lightly," Pizzo said, adding that Microsoft has been moving over internal services dependent upon WCF Data Services to ASP.NET Web API. The amount of work required varies, he said, though teams using Entity Framework or who have already implemented custom data handlers find it an easier task.

"So far, though, the migrations have gone well with the teams enjoying the benefits of more control in how requests are processed, and in the features they expose through their service," Pizzo said.

Most of the work to upgrade WCF Data Services to handle features OData 4.0 has already been done, but some of the features still wouldn't be available because of the "monolithic nature" of WCF Data Services.

Thus Microsoft is moving the project to open source so developers can customize it to suit their own needs. Pizzo asked developers if Microsoft should invest more work in the product to make it more OData version 4.0-compliant before moving it to open source or just do it immediately and let the open source community finish the upgrade work.

Developers weighed in, and many weren't happy:

  • I don't think it really matters when you release the source code. No one is going to care when the owner has stated it's dead. It's a shame I had put a lot of time and investment into WCF Data Services.
  • This is very sad news indeed.
  • Very disappointing.
  • Not telling us until so recently is lame, epic lame.
  • I have no faith in the Web API team prioritizing this issue very high and I fear we end up waiting a long time on the Web API team to catch up -- history shows this to be correct.
  • I know MS wants people to adopt OData and now the people that have are being screwed over. Really not happy with this decision or the thought process that went into it.

Several commenters said Microsoft was erring by focusing on consumer development instead of enterprise development and eroding the trust of front-line coders in the latter camp.

"They need something that can hook to Twitter, Facebook and other Web sources," one commenter said. "In the end, WCF Data Services is just another casualty in Microsoft's pursuit of part of the pie that Google and Apple are getting most of right now."

Others, however, were supportive, and offered their opinions on which way Microsoft should go in open sourcing WCF Data Services, which was the point of Pizzo's blog post.

Pizzo addressed some of the questions and concerns in his own comment on the blog post.

And in response to a query from me, the team provided this:

First, it's important to note that WCF Data Services as it stands today will continue to be supported, including continued support for existing OData 3.0 in WCF Data Services.

We are currently planning to release OData V4 compatibility in WCF Data Services through Open Source, enabling the community to drive the feature set as appropriate.

At the same time, we will be taking the feedback provided to make Web API OData easier to use for scenarios where developers might prefer WCF Data Services today, as well as to make it easier for developers who have invested in WCF Data Services and chose to migrate in order to get the benefits of Web API + OData to do so.

Besides comments in blog posts, much of the feedback provided by developers comes in suggestions on the UserVoice site that Microsoft uses to gather feedback, features suggestions and so on.

A recent entry to the User Voice site is titled "Don't abandon WCF Data Services" and implored the OData team to reconsider the plan outlined in Pizzo's blog post. It had 52 votes today.

Pizzo responded and said the team would consider his feedback "as we prioritize work between enhancing WCF Data Services and providing better support for new features in the WebAPI OData libraries."

I asked the poster of that UserVoice item, Adam Caviness, a senior developer, to expound further. This is what he wrote me:

We were not expecting this news. Based on UserVoice tickets and team blog posts we believed that Microsoft was investing in WCF Data Services alongside of Web API.

In fact, the UserVoice page and Blog were actually named with the WCF Data Services moniker. Microsoft has a tendency to provide multiple offerings of a similar story to their own detriment--just consider why we have the Blend designer in Visual Studio. This is another case where multiple offerings is not a commitment but rather an experiment. Wait, am I a lab rat?

After getting further feedback from the OData Team, we discovered they had been working with internal teams to migrate for some time while we were left in the dark. The OData team also stated twice in posts that "part of our charter is to help Microsoft align behind the OData protocol." I can only gather that Microsoft is making a good technical decision to switch to OData internally, but what meets our needs is more or less a coincidence.

This is a decision they should have contacted the community over and it shouldn't have been mentioned in such a nonchalant [way] in a blog post. WCF Data Services is still listed on MSDN as a viable N-Tier OData strategy so I suppose new unsuspecting developers are even unaware of this direction.

We are investigating the impact of moving to Web API and OWIN. The strategy to reorient Microsoft to a more consumer-centric company is eroding away our confidence in building long-term dependable solutions for this platform.

Me? I can see both sides of this thing. On one hand, you see some of the behavior that leads to such prevalent animosity toward Microsoft on the part of developers who invest in a technology for long-term, line-of-business applications and then see the company going in another direction.

On the other hand, the new direction seems to provide developers with more control and opportunity for customization and might benefit them in the long run. And Microsoft team members are noting the feedback and are answering questions from developers and journalists like me, even though they were quite preoccupied with the just-started Build conference.

Still, I can't help feeling that if I were an enterprise developer as described above, I might hearken back to the immortal words of Johnny Rotten at the end of the Sex Pistols' last concert before the punk band imploded: "Ever get the feeling you've been cheated? Good night!"

What direction do you think Microsoft should go in sending WCF Data Services to open source for OData 4.0 support? Please comment here or drop me a line.

Posted by David Ramel on 04/02/2014 at 1:54 PM0 comments

Entity Framework Gets Code First Model Generation Through Reverse Engineering

Microsoft last week released a minor version update of its popular Object Relational Mapping (ORM) tool with numerous bug fixes and new features, including the ability to reverse engineer an existing database to create a Code First model.

Entity Framework (EF) 6.1 is the first ".x" update since Microsoft open sourced the tool moving to version 6 and moved it to CodePlex. It features tooling consolidation that "extends the ADO.NET Entity Data Model wizard to support creating Code First models, including reverse engineering from an existing database," Microsoft's Rowan Miller explained in an ADO.NET Blog post.

Miller said this new functionality was available in beta quality in the EF Power Tools package, used by the engineering team to preview features under consideration for inclusion in a main release.

Developers can now right-click on a C# project to get a context menu item to "Reverse Engineer Code First," according to Data Developer Center documentation for the EF Power Tools. This command generates a derived DbContext class, "plain-old CLR object" (POCO) classes and Code First mapping classes from an existing database.

Another context menu item is available to "Customize Reverse Engineer Templates," which lets developers customize code generation via reverse engineer T4 templates added to projects.

According to project meeting notes that Microsoft developers now put online, the Code First models that can now be created with the wizard will lack certain features, such as the ability to split tables and entities, and support for complex types and enums, among others.

Microsoft has posted a Code First to an Existing Database video and walk-through where you can learn more about this new feature.

EF6.1 also includes a new IndexAttribute functionality that lets developers specifically identify database table indexes with an [Index] attribute attached to properties in Code First models.

Other enhancements to Entity Framework in version 6.1 listed by Miller include a CommitFailureHandler, public mapping API, performance improvements and many others.

Microsoft engineers such as Diego Vega and Pawel Kadluczka have been busy this week helping developers who reported problems with the new version update in the comments section of Miller's blog.

For more support, Microsoft recommends developers post questions on the StackOverflow site, where six questions had been posted as of today.

For future releases, the EF team's roadmap indicates it's considering "EF Everywhere," described as "a lighter-weight version of EF that can be used across a wide variety of platforms (including Windows Store and Windows Phone)."

Posted by David Ramel on 03/25/2014 at 2:02 PM0 comments

OData Moves Forward

I like tinkering around with OData. And since a certain industry giant has been pushing open JavaScript development over its own vastly superior technologies, I've been trying to wrap my mind around that language using my old friend OData. The jQuery getJSON function helps working with OData, but it's still JavaScript.

(Don't get me started on JavaScript. Actually, I am started, so send me your reasons for hating the language and I'll put them in a future "10 Reasons to Hate JavaScript" post. Should be easy enough.)

Anyway, back to the point: OData is marching smartly forward, with version 4.0 having this week been approved by the OASIS international standards consortium. It has come a long way since being introduced by industry giant Microsoft in May 2011.

Specifically, OData 4.0 and OData 4.0 JSON Format have been approved as standards, with OData 4.0 ATOM being advanced and headed for official sanction.

"OData provides a way to break down data silos and increase the shared value of data by creating an ecosystem in which data consumers can interoperate with data producers in powerful ways, enabling more applications to make sense of a broader set of data," explained OASIS in its announcement.

To me, that reads: "OData is an easy way to access all kinds of interesting public and private data feeds and present the information in cool visualizations and such." You can get lost for hours just exploring the astounding amount of data generated by the federal government, such as, "Job Openings and Labor Turnover Survey," for an example that just happened to catch my eye.

All kinds of goodies are packed into v4.0, like Entity Data Model improvements, better support for distributed services, dynamic schema and an asynchronous pattern for long-running data requests, to name just a few.

Microsoft earlier this month got a jump on v4.0 support, with core .NET libraries, an OData client and a pre-release WebAPI library.

"This is an incredibly exciting time for Open Data, as products and libraries are quickly rev'ing to support OData v4.0," said Microsoft's Michael Pizzo in his announcement that listed all of the goodness I mentioned here and more. "Support for building OData V4.0-compliant clients and servers in Java, as well as consuming OData V4.0 from JavaScript clients, is already underway in the open source Apache Olingo incubator project."

Yi Ding yesterday provided some status updates on the libraries that support v4.0. Ding noted that Java libraries will be part of the Olingo project, due to be completed in the second quarter of this year. A core JavaScript library, Data.js, will also be part of that project. And work is being done to accommodate C++ developers, with a preview client library expected to be available this month. Stay tuned.

Forget this OData stuff. Tell me why you hate JavaScript. Comment here or drop me a line.

Posted by David Ramel on 03/21/2014 at 1:49 PM0 comments

Oracle/Java on Windows Azure and More Data Dev Tidbits

Data development news this week includes the availability of Oracle software and Java on Windows Azure, a service to quickly turn SQL Server stored procedures into RESTful APIs and a database-comparison tool's early support for SQL Server 2014.

First announced in a preview last June by Satya Nadella (who's been in the news himself a bit recently), the Windows Azure/Oracle collaboration was made official with a general availability announcement yesterday. Java developers can now work with Oracle's database and WebLogic Server on licensed virtual machine (VM) images.

"We'll also work together to add properly licensed, and fully supported Java into Windows Azure -- improving flexibility and choice for millions of Java developers and their applications," Nadella said last June. "Windows Azure is, and will continue to be, committed to supporting open source development languages and frameworks, and after today's news, I hope the strength of our commitment in this area is clear."

Well, one thing that's clear is that Nadella now has much more gravitas to see that commitment through.

As Jeffrey Schwartz reported yesterday on, Java support on Windows Azure was previously available through the open source OpenJDK. "For those who wanted to use Oracle's Java license, the partnership offers a fully licensed and supported Java on Windows Azure," Schwartz explained.

In other news this week, Espresso Logic Inc. announced developers can now quickly create RESTful APIs for stored procedures with its cloud-based platform, a "reactive service for SQL data." Espresso 1.2 supports SQL Server version 2005 and later, along with Windows Azure SQL Database, Oracle and MySQL.

"Using the new functionality, developers can quickly create mobile applications that leverage a company’s investment in stored procedures," the company said. "The Espresso Designer exposes stored procedures as REST/JSON resources while the supporting infrastructure is handled by the Espresso service."

The announcement is among many that have recently highlighted the growing importance of RESTful APIs in mobile app development.

Meanwhile, just today, xSQL Software announced it was getting an early jump on SQL Server 2014 support by its SQL Server schema compare tool.

Enhancements to version 5 of the xSQL Schema Compare and Data Compare tools include "all new and improved SQL Server 2014 objects such as memory-optimized filegroups, memory-optimized tables, hash and range indexes, columnstore indexes, memory-optimized table types, natively compiled stored procedures, primary and secondary selective XML indexes, XML index and namespace paths," the company said.

SQL Server 2014, of course, is coming soon.

What do you find new and exciting in the world of SQL Server development? Weigh in here or drop me a line.

Posted by David Ramel on 03/14/2014 at 12:07 PM0 comments

No Stopping Microsoft Haters

I feel for you, Scott Hanselman, I really do. I completely understand your wanting to address the irrational vitriol constantly directed toward Microsoft, especially by the developer community. But your blog post, "Microsoft killed my Pappy," won't alleviate anything. It just won't work. As the saying quoted by one commenter goes, "haters gonna hate."

I've got to admit, I used to vaguely dislike Microsoft. For all the usual reasons: big, monolithic, evil empire out to enrich itself through any means possible while trampling over the little guy, dictating to rather than listening to customers, and on and on.

Then I got this job and really began to look at the company. It's no more evil than any other corporate behemoth. It's doing some good things in open source development (Hanselman's bailiwick) and in many other areas. It produces first-class dev tools and listens to customers, changing things they don't like. Witness the silly ALL-CAPS fiasco for Visual Studio menus and much more important things like early access to Windows 8.1 for developers. Microsoft is a bunch of generally well-meaning people simply trying to do their job, just like you and me. As Hanselman pointed out, the company isn't nearly organized enough to be so evil.

But haters gonna hate. Of course, Hanselman is a much smarter guy than I am, and he knew that. Maybe he just got fed up with the unfairness of it all and had to address it. More likely, he wanted to slowly chip away at the issue and stir up some discussion. Which he surely did, with more than 250 comments to his post as I write this.

"I didn't work for The Man when all this [antitrust action and other things] went down, and I was as outraged as you were 20 years ago (assuming you were alive)," Hanselman said. "But now, I've been here pushing open source and the open web for over 5 years and things are VERY different."

He roused comments from a fair share of sympathizers, but, this being the Web, comments quickly deteriorated, with everybody attacking everybody. At one point, Hanselman had to step in with the comment, "Good discussion folks! Let's do keep the language clean and constructive and avoid hyperbole," after one profanity-laden post.

Readers came up with dozens and dozens of reasons to hate Microsoft, including killing off their favorite products (speaking of which: Keep Silverlight Alive!). One reader seemed to sum up the comments from that camp: "MS hate is well-deserved."

I don't think so. I think Microsoft is an easy target. Back when I vaguely disliked Microsoft, I used to work for a well-known, nationwide print/Web tech publication. After I left and began working with Microsoft technologies more, I saw a shocking pattern of bias against Microsoft at that media outlet. I couldn't believe it. One of the first things you learn in Journalism 101 is to be objective, no matter what your personal feelings (or maybe, these days, no matter how many clicks you're trying to get). That's gone out the window at that former company, and they might not even be aware of it.

I actually thought of collecting a bunch of headlines and lumping them together so the bias is readily apparent. The headlines with negative connotations must outnumber the others 10 to 1. One guy in particular has an absolute knack for taking any kind of news about Microsoft and putting a negative spin on it--even when the company does something (arguably, in my opinion) good. It's amazing. I felt like pointing this out to the editor in chief, a friend of mine, but I don't have time and anyway I quickly realized it would be useless. Haters gonna hate, and "news" organizations gonna chase clicks (yeah, me too). Everybody hates Microsoft? Feed the frenzy and watch the clicks climb in your Web site analytics tool. I can understand this attitude in opinion pieces, like this, but it's a much more insidious and dangerous practice when it seeps into so-called "objective" news articles. It demeans the company and my profession even more so.

Speaking of clicks, I garnered a few myself with an article titled "/* Microsoft should go to hell...*/: Developers Rip Microsoft in Source Code." I found it fascinating that so many developers took pains to attack Microsoft in source code comments. Who's going to read those? Hardly anyone, even after the code search tools came along. The article sparked a lively conversation on Hacker News. A much more recent post titled, "Ask HN: Why the Microsoft hate?" did even better, with 540 comments.

Over on Slashdot, a post titled, "Why Does Everyone Hate Microsoft?" garnered an incredible 1,540 comments (Hanselman's article isn't doing too badly over there either, with 721 comments just four days after publication).

I don't really know the answer to these questions, but they surely strike a nerve, with developers especially.

So please excuse my going off-topic with this deviant diatribe, data developers. Hanselman's article just exemplified something I think about every time I see Microsoft attacked in the "objective" press. But, like so many things that confound me these days, there's no stopping it.

Let's go positive! What do you LOVE about Microsoft? Comment here or drop me a line.

Posted by David Ramel on 02/25/2014 at 1:15 PM0 comments

Visual Basic .NET Sees Popularity Hike

OK, raise your hands: Do you do your data programming in Visual Basic .NET? If I was a speaker at a development conference and asked that question, I'd expect to be squinting to see the results.

Yet this month's TIOBE Programming Community Index, which measures the popularity of programming languages, reports that Visual Basic .NET has cracked the top 10 for the first time ever, right between JavaScript and Transact-SQL (the latter of which, by the way, was recently named by TIOBE as "programming language of the year").

Though not specific to data-related programming, the TIOBE index measures popularity by counting the number of developers using a language, courses offered, search engine results and more.

Visual Basic .NET saw an increase of 0.79 percent, the fourth-highest percentage gain among the top 20 languages listed. That moved it from No. 12 last February to No. 10 this month.

Here's TIOBE's take on the news:

This is quite surprising for two reasons. Visual Basic .NET is the successor of Microsoft's well-beloved classic Visual Basic 6.0 version. Since Visual Basic .NET needed to run on Microsoft's .NET platform, the language has changed drastically. Many software engineers refused to migrate to Visual Basic .NET. For this reason, Visual Basic .NET has been criticized through the years. The other reason why this is surprising is that Microsoft seemed to slow down further development of Visual Basic .NET. For example, the latest Visual Studio version 2013 doesn't contain any new Visual Basic .NET language features.

The index shows "(Visual) Basic" at No. 7, even though it suffered a percentage drop. The "classic" Visual Basic has fallen considerably since its peak in April 2004 and is lately trending steeply downward. In fact, Visual Basic .NET, which debuted on the index in September 2010, has been climbing and it looks like the two could intersect in the next few months.

It's hard to make similar comparisons to TIOBE's rankings with other such programming language popularity tools because many don't distinguish between classic Visual Basic and Visual Basic .NET or don't show historical trends. Anyway, for what it's worth, examinations of other sites, of varying formats and publication dates, shows the following. The PYPL PopularitY of Programming Language Index shows Visual Basic at No. 10 in its chart. shows Visual Basic at No. 16. On the Transparent Language Popularity Index, Basic (no "Visual") was No. 5 last July. The RedMonk Programming Language Rankings: January 2013 site shows Visual Basic at No. 19. You can also fool around with tools at TrendySkills and Google Trends.

What do you make of this rise in popularity? What do you think of Visual Basic .NET? Comment here or drop me a line.

Posted by David Ramel on 02/21/2014 at 11:58 AM0 comments

Microsoft's Data Direction: the Cloud

A recent blog post from Microsoft's data guru titled "What Drives Microsoft's Data Platform Vision?" provided a clear answer: the cloud. Well, duh.

No news there. For quite a while now, at Microsoft, it has been: "The cloud is the answer. What's the question?" Specific details data developers might be interested in, however, were scarce in the post by Quentin Clark, corporate vice president of the Data Platform Group. There was a lot of stuff like, "After the delivery of Microsoft SQL Server 2012 and Office 2013, we ramped-up our energy and sharpened our focus on the opportunities of cloud computing."

I found myself trying to read between the lines. Was there any new information here? Any subtle clues? Any news by omission? I became a Microsoftologist.

To explain: In the old days, before The Wall came crumbling down, news coming out of the Soviet Union was so skimpy it fostered an analysis technique called Kremlinology, wherein Western strategists tried to glean insights about the direction of the Evil Empire by noting little details like who stood next to whom in parade reviews and combing through speeches for hidden clues.

What struck me most: Where's the Big Data? The rest of the data development world is going crazy about Big Data, but in this post, not so much. It was cloud, cloud and more cloud. My Microsoftology revealed 13 references to "cloud" and seven to "Windows Azure." The total number of "Big Data" references? Two. But there was one reference to "HDInsight" and five to "Hadoop." So, total score: cloud, 20; Big Data, 8.

I also noted in other news that Microsoft development rock star Scott Guthrie (ScottGu) was reportedly named to be the new head of the cloud division, replacing the new CEO, Satya Nadella.

Hmm. Maybe if you're a Microsoft data developer, you should be "Getting Trained on Microsoft's Expanding Data Platform" by taking classes such as "Platform for Hybrid Cloud with SQL server 2014" and "Windows Azure SQL Database" rather than "Getting Started with Microsoft Big Data" and "Big Data Analytics," as featured in another blog post yesterday.

Then again, another Microsoft blog post last week noted: "Microsoft to Join the Big Data Conversation as an Elite Sponsor at Strata, Santa Clara, CA." Just joining the conversation? Maybe it's just late to the party.

What's your take on the future of data development on the Microsoft stack? Comment here or drop me a line.

Posted by David Ramel on 02/07/2014 at 12:18 PM0 comments

Data Survey: Microsoft Fending Off Challengers, for Now

A new survey of database developers and other professionals shows Microsoft is maintaining its lead in the Relational Database Management Systems (RDBMS) and data warehousing arenas, but faces challenges from newcomers in Big Data and other markets.

Conducted by Progress Software Corp., the "Progress DataDirect 2014 Data Connectivity Outlook" survey purported to reveal "the rising stars in the database constellation." Developers constituted the largest group of respondents (36 percent of 300 existing customers surveyed), followed by CXOs and other management types.

For the RDBMS and enterprise data warehouse markets, respondents were asked about their data technologies currently in use and those expected to be implemented within the next two years. Although Microsoft (SQL Server) and Oracle unsurprisingly took the top two positions in current and projected usage in the RDBMS market, the survey "projects significant growth for emerging alternatives such as the community-developed MariaDB as well as the SAP HANA in-memory platform, over the next two years," Progress said this week.

In the enterprise data warehouse world, the top three technologies currently in use were again no surprise: SQL Server, Oracle and IBM DB2. However, all three were projected to show lower numbers in two years, while the current No. 4, Teradata, was expected to see higher usage. Amazon Redshift will reportedly show the biggest percentage increase in adoption, but is still expected to be in use by only slightly more than 10 percent of respondents in two years. SQL Server usage is expected to drop from being used by about 58 percent of respondents today to about 48 percent of respondents in two years.

Perhaps more of a surprise, Microsoft HD Insights was listed as the No. 2 Hadoop provider, following overwhelming market leader Apache's open source distribution, listed by more than 45 percent of respondents. Cloudera placed a close third, followed by Oracle DBA, Amazon EMR, IBM BigInsights, Hortonworks, MapR and Pivotal HD. This question asked respondents only what distribution they currently used or planned to use in the next two years, so there was no indication of growth over that period.

"With open-source Apache's low-cost of entry propelling its lead in the market, one can expect other big players in Big Data to further iterate their own unique value and perspectives when it comes to data storage within Hadoop databases like Hive, HBase and Cassandra," the survey report said. "Future competition from many of the large vendors may begin to change market distribution, but no significant changes are foreseen."

Another question concerned usage of NoSQL, NewSQL and non-relational databases, asking only which technologies were currently used or supported. Here, MongoDB held a large lead, used by nearly 40 percent of respondents, with SQLFire, Cassandra, HBase and CouchDB/Couchbase rounding out the top five of the 14 total products listed. reported a huge lead in respondents answering the question: "Which [Software as a Service ] SaaS applications do you or your customers currently use or support in your applications?" It was the choice of more than 40 percent of respondents, while Microsoft Dynamics CRM Online came in second, listed by more than 20 percent, followed by SAP Business ByDesign, and Intuit QuickBooks.

Progress said its first survey of this type "shows that while established vendors still hold significant share, a new set of rising stars--many of them lower-cost alternatives--are emerging in the data source world."

What do you think of the future of the database development landscape? Comment here or drop me a line.

Posted by David Ramel on 02/03/2014 at 11:59 AM0 comments

What Languages Should You Learn in the New Web/Mobile World of 2014?

I've noted before how data-driven developers in general and SQL gurus in particular are pretty well set in terms of salary and job security. So I was curious how database skills fared in responses to a recent question: "It's 2014--Which New Technologies Should I Learn?"

An anonymous reader's "Ask Slashdot" posting on Wednesday read thusly:

"I've been a software engineer for about 15 years, most of which I spent working on embedded systems (small custom systems running Linux), developing in C. However, Web and mobile technologies seem to be taking over the world, and while I acknowledge that C isn't going away anytime soon, many job offers (at least those that seem interesting and in small companies) are asking for knowledge on these new technologies (Web/mobile). Plus, I'm interested in them anyway. Unfortunately, there are so many of those new technologies that it's difficult to figure out what would be the best use of my time. Which ones would you recommend? What would be the smallest set of 'new technologies' one should know to be employable in Web/mobile these days?"

I was so curious I combed through more than 370 comments to total up and compare the "technologies." Obviously, that's a broad term and could (and apparently did) mean just about anything, so I just focused on programming languages (as opposed to, say, "Learn to lie and [BS] with a straight face"). And I wasn't alone in wondering what constituted a "new technology."

Of course, this being Slashdot, the readers branched off on all kinds of bizarre tangents. It's amazing how these people can take the most insignificant, meaningless aspect of such a question and absolutely beat it to death. It's often pretty darn funny, though.

Anyway, a lot of Slashdot readers know their stuff, so I was interested in what they had to say, regardless of the wide range of possible interpretations of the question. My sampling is in no way scientific, or a real survey or even reliant upon any kind of reproducible methodology. I simply tried to total up the language suggestions I found in each of the comments. I didn't subtract votes when a suggestion was hammered by other readers with the inevitable vicious insults and snarkiness (some things will never change).

I guess the results were fairly predictable, but I was kind of disappointed in how database technologies in general ranked.

What was completely clear is that the overwhelming favorite is ... you guessed it: JavaScript. A rough grouping (not counting a lot of people who said just stick with C) looks something like this:

  • First: JavaScript
  • Second: Java
  • Third: PHP, HTML(5)
  • Fourth: Objective-C, Python
  • Fifth: Ruby, SQL, C++, C#
  • Sixth: HTTP, ASP.NET, CSS

Several dozen more languages were suggested in smaller numbers.

Being the resident Data Driver bloggist at Visual Studio Magazine, I was disappointed to see SQL so far down the list. Even totaling up all the other database-related languages, such as MySQL, SQLite, MongoDB and so on, wouldn't result in that impressive of a number (I stopped counting these when I realized none would total more than a few votes).

A couple of comments might shed some light on the prevailing attitudes out there. One commenter wrote: "RDBMSes are going to die, so learn how to interact with one of the major NoSQL databases. Most bleeding-edge: Titan and Neo4J, both graph databases."

Another wrote: "Some SQL is very useful but you don't need to be an expert--any serious Web development team will have a database expert who will do the DB stuff, you just need enough to code up test setups, prototypes and to talk to the DB guy."

I don't know exactly why "the DB guy" is separate from the rest of the Web dev team, or why the original poster couldn't be "the DB guy," but whatever.

The question was limited to the Web/mobile arena, remember, so it's not totally disheartening. I mean, there is this little thing called Big Data happening, and vendors are jumping all over themselves trying to come out with applications and packages and such to let the SQL guys and other "DB guys" join in the fun along with the Hadoop specialists and data scientists. But I guess nobody will be doing any Big Data stuff over the Web or with a mobile device.

And SQL didn't fare too badly in more broad examinations of this topic, earning a spot in "Top 10 Programming Languages to Know in 2014" and "10 Programming Languages You Should Learn in 2014." Also, of course, Transact-SQL was named "programming language of the year" for 2013 by TIOBE Software.

What do you think? What would be your top suggestions for staying current in this new world? Comment here or drop me a line.

Posted by David Ramel on 01/24/2014 at 8:16 PM0 comments

Transact-SQL Named 'Programming Language of the Year' for 2013

Maybe it's not the sexiest programming language, but SQL continues to be relevant. In fact, TIOBE Software, which publishes a TIOBE Programming Community Index gauging the popularity of programming languages, named Transact-SQL the language of the year for 2013.

This "award" further emphasizes the importance of competency in SQL. I earlier wrote about how SQL gurus and other database-related programmers enjoyed excellent job security and how SQL Server developers were in high demand.

That's the good news. The bad news, according to TIOBE, "It is a bit strange that Transact-SQL wins the award because its major application field, Microsoft's database engine SQL Server, is losing popularity. The general conclusion is that Transact-SQL won because actually not much happened in 2013."

Not much happened in 2013? Wow, talk about strange. Has TIOBE heard of a little thing called Big Data?

Anyway, following Transact-SQL in popularity gains were Objective-C and F#. Objective-C had been the "language of the year" for the previous two years.

Microsoft fared well in other aspects, too, even regarding the much-maligned Windows Phone platform. As TIOBE wrote: "As we have seen the last decade, programming language popularity is largely influenced by external trends. The most important ones at the moment are mobile phone apps and web development. Android (mainly Java) and iOS (Objective-C) are the major mobile platforms, while Windows Phone (mainly C#) is catching up."

In index rankings, Transact-SQL went from No. 22 in January 2013 to No. 10 in January 2014. Otherwise, the rankings stayed the same for the top eight positions: C, Java, Objective-C, C++, C#, PHP and Python. The only other move in the top 10 was JavaScript going from 10th to the 9th spot.

In other attempts at ranking the popularity of programming languages, SQL was No. 12 in a list developed by last October. Meanwhile, Python garnered the "language of the year" prize in the Popularity of Programming Language (PYPL) index, which measures how often respective language tutorials show up in Google searches. No variants of SQL made the top 10. TIOBE said its ratings "are based on the number of skilled engineers world-wide, courses and third party vendors."

In Google Trends, searches for "SQL Programming Language" held fairly steady throughout 2013, except for a strange dip right at the end of the year.

How do you feel about the importance of keeping your SQL skills honed? Do these popularity rankings mean anything at all? Comment here or drop me a line.

Posted by David Ramel on 01/16/2014 at 11:23 AM0 comments

SQL Continues to Crash the Big Data Party

Regardless of the future of the Microsoft ecosystem (and those latest quarterly numbers should slow the naysayers some), data developers can rest easy knowing their SQL Server skills are transferable in the New Data Order.

The latest example is yesterday's open sourcing of a new distributed SQL query engine for Big Data developed by Facebook, called Presto.

It was designed to improve upon existing solutions for Big Data analytics such as Hadoop MapReduce and Hive, Facebook's Martin Traverso said in a post announcing the move to GitHub. "Presto is a distributed SQL query engine optimized for ad-hoc analysis at interactive speed," Traverso said. "It supports standard ANSI SQL, including complex queries, aggregations, joins and window functions."

Traverso said Presto has provided performance gains of up to 10 times more than equivalent Hive/MapReduce tools in CPU efficiency and latency for most queries. While it doesn't run on Windows, "It currently supports a large subset of ANSI SQL, including joins, left/right outer joins, subqueries, and most of the common aggregate and scalar functions, including approximate distinct counts (using HyperLogLog) and approximate percentiles (based on quantile digest)," he said.

Yes, SQL isn't going anywhere. It has withstood challenges in one form or another from other Relational Database Management Systems such as Oracle, branch movements such as MySQL, hybrids like the NoSQL movement and so on. The Big Data onslaught seemed to be stealing much of its mindshare, but the pendulum is swinging back. The problem was that the specialized Hadoop-based solutions often proved too cumbersome to quickly and efficiently glean meaningful analytics from the vast new data stores.

"This enormous knowledge gap in accessing Big Data in Hadoop has prompted an avalanche of vendors to offer SQL-on-Hadoop solutions, which increase the accessibility of Hadoop and allow organizations to reuse their investment learning in SQL," stated a Gigaom report titled "Sector RoadMap: SQL-on-Hadoop platforms in 2013."

"SQL is widely known by most business analysts," the report continued. "Many nontechnical staff without a programming background can write SQL and use traditional business intelligence (BI) tools like Tableau, MicroStrategy, and Business Objects to query data."

Further evidence of SQL's solid positioning came in a recent presentation by Roger Magoulas, research director at O'Reilly Media, at the Strata Conference + Hadoop World event. He spoke about "the state of data science as a profession." An O'Reilly salary survey conducted last year reported that the top tool in use by the responding data scientists was SQL. "I guess it's not a surprise ... we heard some of the other speakers talk about it ... that SQL is still the top thing being used," Magoulas said. His accompanying slide proclaimed "SQL Rules" and indicated 71 percent of respondents reported it as their data science tool of choice. Hadoop was a surprising No. 5 at 35 percent. SQL users also fared well in the salary level findings, but averaged below the far-fewer number of Hadoop specialists.

You can expect to soon hear about more SQL-related Big Data initiatives, joining high-profile efforts such as Teradata's Enterprise Access for Hadoop; Cloudera's Impala; IBM's Big SQL Technology Preview; Hortonworks' Stinger; and many, many more. Stay tuned.

What do you think of the SQL-on-Hadoop and other SQL-related Big Data technologies? Comment here or drop me a line.

Posted by David Ramel on 11/07/2013 at 12:50 PM0 comments

Upcoming Events

.NET Insight

Sign up for our newsletter.

I agree to this site's Privacy Policy.