Practical .NET

Simplifying Applications by Implementing Eventual Consistency with Domain Events

Implementing Domain Events can dramatically simplify your application while also making it more scalable. Here are some of the implementation options available to you in the .NET environment.

• By Peter Vogel
• 10/13/2015

The process of notifying another domain does require some level of cooperation between the application and the domain being notified: The application must provide the information that the other domain requires. At the very least, that information will include a code indicating the kind of transaction (customer update, salesorder deletion and so on), some sort of priority flag, and the data that uniquely identifies the data that makes up the transaction (for example, a customer number, a sales order number).

For domains within the same organization the domain can use the keys in the message to retrieve the relevant transaction data. Where the application and the domain are in different organizations, because the transaction's information isn't available in the other organization, the details of the transaction will also usually be included in the message. An option here, however, is to have the application in the other organization fetch the data from the original application's company when the domain finally gets around to processing the message (this ensures the domain always has the most up-to-date information). In that scenario, the message can go back to just including the key data.

To support integrating the application and the domain responsible for the updates, the team responsible for the domain will need to define a message format that the team building the application can implement when generating the message. This could be a class interface, a WSDL contract or the documentation describing a RESTful service that an application must call.

Sending Messages
Microsoft Message Queue (MSMQ) provides a great storage location for messages for both the application and the domain. Among other benefits, MSMQ guarantees the survival of all messages in the event of a server shutdown and MSMQ messages are transactional (if the application rolls back a transaction that includes writing to a queue, the message is automatically deleted as part of the rollback). Within an organization, it might be possible for an application simply to write its message to the queue that the other domain reads (MSMQ supports writing to queues on other servers).

Using a queue also allows the application to get out of the business of sending messages. After the application writes to the queue, another application (I'll call it the "sending application") can be responsible for sending messages to the domain, updating the message log, and deleting the message when the acknowledgement is finally received from the domain. The sending application can simply process messages as they're received or can give precedence to high-priority messages (or to messages going to specific domains -- those belonging to Very Important Business Partners, for example).

With a sending application in the message pipeline, the information about where the message should be sent should be stored with the sending application rather than with the application that creates the message. This centralizes all the communication information in one place where it can be updated as destinations change. This sending queue is also relatively easy to monitor to determine what messages are "stalled" because of problems with the sending application or with a specific domain not accepting messages.

Receiving Messages
At the domain end, the message may be processed immediately, written to a queue/database table for later processing or some combination of those options. For example, my credit-card company seems to process the amount spent immediately (subtracting it from my "amount available to spend") and defers all other processing until later. If the receiving process simply writes messages to another queue it can process messages quickly, making this part of the pipeline extremely scalable.

Eventually, however, messages received by the domain must be processed. The obvious solution is to have individual applications scan the queue looking for messages they need to process. This second option is appropriate when messages are processed in a batch at specific intervals (after the end of the business day, for example), and by only one application. If multiple applications can process the same message, some sort of process for deleting messages will need to be implemented.

Where you want to reduce the delay between when a message is sent and processing occurs, a better design is to have a single application that processes queue of pending messages. This application can be hardwired to call specific processing for specific messages and then delete the message after all processing is complete.

An alternative to hardwiring the message processing application is to use a subscription model. In this model, a record is added to a database table indicating what kinds of messages specific applications should process. The message processing application reads that list and calls the specified application when relevant messages show up. Among other benefits, this design allows a single message to trigger processing in several applications before the message processing application finally deletes the message. Malin de Silva has described the Command Pattern, which can be useful when providing a standard way for the message processing application to notify the subscribed application.

In a variation on the subscription model, a "preprocessor" reads incoming messages, matches it up with the subscribed applications and generates a queue of matched messages and subscribers (this application needs to be speedy enough that it can always keep up with the incoming messages). The message processing application now reads this queue of matched message/subscribers, passing each message to the specified subscriber.

Regardless, the message processing application doesn't delete any message until the subscribed applications indicate that they have successfully completed processing the message. A monitor process should allow you to scan the queue to look at the state of the messages in the queue.

Dealing with Stale Data
There is one unavoidable problem with Domain Events: Applications might be working with stale data. It's entirely possible that some other application in another domain has updated some data and those updates are in the "eventual consistency" queue. It's also possible that, at busy times, the message processing application may fall behind and messages may sit in the queue for a significant period of time.

For extremely critical processes, applications can scan the queue looking for relevant transactions before processing. For example, a shipping program might check for change-of-address requests for any customer who's order is about to be shipped; a payment-authorization program might scan for pending changes to a customer's credit limit or expenses. Effectively, applications "jump the queue" instead of waiting for the message processor.

If you're going to allow multiple applications to scan the queue of "pending requests," you might be better off keeping your "eventual consistency" queue in a database table where applications can search by type of transaction and related tags (customer number, for example) more quickly than they can scan an MSMQ queue. To keep these tables small so that they can be searched quickly, it might be worthwhile to dedicate tables to specific kinds of transactions.

In a subscription model, where you allow applications to "jump the queue," the application will need to flag any pending transactions as "already processed" to prevent the message-processing application from processing the message again. A preprocessor that generates a queue entry for each message/subscriber pair simplifies this -- when an application jumps the queue, the application simply flags its entry as "processed."

As you can see, you have multiple options when it comes to implementing Domain Events. You'll need to decide which ones make sense to you and your organization. Your goal, however, is always to create an application that you can actually understand, monitor, and maintain. If you can do that, you just might break out of the CRAP cycle.