DevDisasters

Doubly Precise VB.NET Rounding Issues Abound

You'd think that rounding should be simple, but why is it causing Ryan so much trouble? You can thank his managers for that.

• By Mark Bowytz
• 09/01/2015

In a big organization, like the one where Ryan works, management is understandably a little touchy about throughput on their systems. Therefore, Weekly Operational Pulse Reports are e-mailed out every Monday at 2 a.m.

The output isn't anything fancy, and it doesn't have to be. The reports are monospaced, tabular lists that cover the daily transactions per minute (TPM) for the last three weeks and, including the descriptive error messages and descriptive details, are a few pages in size when printed out.

The dirty secret of it all? Nobody ever really looks closely at the numbers. What it comes down to is that while management is absolutely concerned about transactional volume, what they really strive for is uniformity and predictability.

Let's say you have a day with zero records per hour. Yeah, that'll be noticed. Or, if you're like Ryan, and you have a two-week streak of 40.00 records processed per minute only to be interrupted by a Thursday with 39.99 records, you're going to have eight different bosses asking questions. After all, it's their job to ask tough questions before upper management ahead of those further up the food chain start asking.

To kick things off, Ryan started at the source: a database table with more than 10 million rows that tracked every transaction. He ran the query in the TSQL procedure that's behind the weekly e-mail and dumped that into Excel. Everything checked out -- every day, including that odd Thursday, rounded to 40, but there was one detail worth noting.

What gives? Unfortunately, the answer had to be in the VB.NET code behind the actual rounding itself. The console application behind the e-mail (PulseReport.exe) written circa 2008 and had survived two server migrations apparently had a reputation for being a little bit quirky.

So, when Ryan found the function, it was pretty much exactly as he had expected:

Public Function myRound(dblNum As Double, iDec As Integer) As Double

  Dim dblFac As Double
  Dim dblTmp As Double
    
  dblFac = 10 ^ iDec
  dblTmp = dblNum * dblFac + 0.5
  myRound = Int(dblTmp) / dblFac

End Function

Now, the oddball number that needed to be rounded was 39.995, and it needed to be rounded to two decimal places. Human logic dictates that the result should be 40.00, but after running a test program, Ryan found that the rounding routine was returning 39.99.

In fact, upon further testing, the values of 32.995, 33.995, 34.995, 35.995, 36.995, 37.995, 38.995 and 39.995 all produced the incorrect results.

What made this a head-scratching moment was when other numbers like 11.995, 49.995, 39.994 and 39.996 all produced the correct values (12.00, 50.00, 39.99 and 40.00, respectively).

Internet spelunking into the issue yielded plenty of info about Banker's rounding (rounding to the nearest even number), Asymmetric Arithmetic rounding (positive numbers away from zero, negative numbers round toward zero) and Symmetric Arithmetic rounding (negative and positive numbers round toward zero), and juicy tidbits about how the Round function is implemented in the various Microsoft products (along with people who seemed to be in the same boat as Ryan with rounding woes of their own). He also found that Double precision rounding issues in the Microsoft .NET Framework were known to cause troubles.

But he found nothing that could explain definitively why rounding 39.995 produced 39.99 instead of 40.00 in this code.

To correct the situation (after all, the bottom line is important), Ryan changed the code to:

myRound = Math.Round(value+0.0005, 2, MidpointRounding.AwayFromZero)

With that simple change, Ryan saw all of the results were as expected.

Sometimes, trying to explain misbehavior isn't quite as important as just making something work the way it should -- or at least how managers expect that it should.