Practical .NET

Simplifying Applications with Eventual Consistency

Stop trying to make everything work all at once. Instead, use Domain Events to make applications simpler, more scalable and easier to maintain -- and to defer updates until you can't avoid making them.

• By Peter Vogel
• 10/19/2015

When I think about data integrity, I think about banks. Banks are organizations that absolutely, positively must have 100 percent accuracy with their inventory (which is, after all, your money). Yet, when I look at my bank statement online, I notice that it's not up-to-date. Some transactions that I've performed aren't reported yet or, if they are reported, are missing key information that doesn't show up until the next day.

The same is true of my online credit-card statement where my "available credit" and "amount spent" very seldom add up to my credit limit. Here, also, I notice that my list of transactions often lags behind reality, especially for purchases below a certain threshold. I also notice that, even when one of my purchases is reported as soon as I make it, detailed information about that transaction doesn't show up until later, often not until the next business day.

There are more discrepancies between reality and my online credit-card report: My "amount available to spend" should, I think, be the difference between the total cost of my listed transactions and my credit limit. Often, my "amount available to spend" is less. Obviously, my "amount available to spend" is being adjusted downward by purchases I've made that aren't displayed on the transaction list … at least, not listed yet.

One last thing: I've also noticed that my "amount available to spend" will jump up even though I haven't sent my credit card company any money. Behind the scenes, I assume, various credits are being applied to my account to "fix up" discrepancies.

Obviously, my credit-card company (and my bank) has decided they don't need my data to be completely up-to-date. While not ideal, my credit card company has obviously asked two crucial questions before trying to build a completely consistent application that fully integrated all of my activities: "What's the risk?" and "What's the worst that could happen?" The worst that could happen is that I manage to sneak in some more purchases after reaching my credit limit.

The credit card company manages that risk by insisting on being up-to-date about the amount I've spent (and, even then, they only seem to care about "large" purchases). In fact, just to make sure I don't overspend, sometimes the amount I spend gets subtracted twice (that's a problem referred to by the term "idempotent") and has to be fixed up later through those credit adjustments I see occasionally.

By aggressively updating information that limits my spending (while deferring other updates), the credit-card company manages its risk of giving me too much money. This process could go wrong. It's possible that, if I'm close to my limit, and want to make a large purchase, I might be unfairly denied credit (fallout from the idempotent problem). However, I imagine, a call to customer service would allow a human being to review my expenditures and delete any obvious duplicates and let me make that purchase. And, of course, that's more my problem than the credit-card company's.

Furthermore, while there are discrepancies, they only occur during a "business day." Typically, by "the next business day," my account accurately reflects my reality. The longest time that my account won't be consistent will be three days, from the close of business on Friday to the start of business on Tuesday morning after a long weekend.

These companies -- with their requirement for 100 percent inventory control -- have decided that the costs associated with keeping data up-to-date aren't worth the benefits. You could learn something here when it comes time to build your applications.

Avoiding CRAP
I've referred before to the Create, Repair, Abandon, rePlace (CRAP) cycle through which many applications go. What starts off as a beautifully designed application descends, through a history of maintenance, into an application so badly architected and so complex that everyone is afraid of changing it. Often these are "enterprise-wide applications." The idea behind an "enterprise-wide application" is that it's possible for mere mortals to create applications that span many parts of the organization and pull all of those parts together into a smoothly functioning whole.

Domain-driven development (DDD) says that just ain't so. Instead, DDD predicts that, as parts of the organization change in different ways at different times and at different speeds, those "enterprise-wide applications" become shackles rather than foundations. Much of the complexity in creating an "enterprise-wide application" comes from an effort to keep all information current everywhere all the time, forcing developers to attempt to coordinate transactions across multiple databases and applications (the infamous "two phase commit").

DDD says: Give up. Now.

DDD suggests that the right answer is to create applications that support smaller domains within the organization and have these domains communicate with each other to coordinate updates. Ideally, these domains are loosely coupled so that they can evolve relatively independently. But, with that loose coupling of simpler applications, comes a cost: All of your data won't be up-to-date all the time -- some of it will be in transit from one domain to another and some of it will be waiting processing in the "other" domain. The major banks and credit-card companies have decided they can live with this cost.

Domain Events
Accepting this cost means that you divide transactions into two groups. First are those transactions that implement updates that require immediate consistency (for example, notifying the credit-card company that I've spent $150). Those transactions are represented by what DDD calls an aggregate and are typically performed within an application. Applications that concentrate on the small number of updates requiring immediate consistency are going to be significantly simpler than any "enterprise-wide application."

By definition, any update performed by another domain falls into the second category: Those transactions that require eventual consistency (applications can also use eventual consistency internally, of course). The details of my expenditure, ensuring that transactions are idempotent, and reporting small purchases are all, for my credit-card company, transactions whose updates only need to be eventually consistent. For updates performed by another domain, the only thing that any application has to do is send a message to that other domain requesting the updates. This not only makes the application simpler, but because you're offloading work from your application to another computer, your application processes faster and can support more users – and your application becomes more scalable.

To ensure things are eventually made consistent, the application sending the message must continue to send the message until it receives an acknowledgement from the other domain that the message has been received. Obviously, in the real world, repeatedly sending the message might result in the message being received and processed multiple times (the source of the idempotent problem). It's the receiving domain's responsibility to recognize and handle multiple receipts.

My credit-card company obviously has decided that it won't try to handle the idempotent problem as messages are received but, instead, counts on an overnight process that checks for those problems and applies credits to resolve it. I can testify that the solution isn't perfect, at least with my company: I recently ended up talking to someone at customer service about a hotel charge that had been applied twice. But notice that, if the process fails, it's a problem that prevents me from exceeding my credit limit. I suspect my credit card company is fine with that "problem."

Eventual consistency leads to what DDD calls Domain Events: Applications "raise an event" by preparing a message and ensuring that the message has been received by the other domain (though keeping a log of messages sent and acknowledgements received is probably a good idea for the application, also). And, the good news is, when it comes to actually implementing Domain Events in the .NET world, you have lots of options in how to do it!