Practical .NET

Managing and Enforcing Message Formats for Services

You can dramatically simplify life for developers creating services and their consumers by following three rules for designing messages and then enforcing your message formats with JSON Schema.

• By Peter Vogel
• 02/27/2018

In a previous column, I discussed a service-based application that provided a terrible user experience and limited its own scalability because of poor message design. While I didn't mention it in that column, I suspect that the service's message design would also limit the ability to exploit those services to create new applications.

That article demonstrated the three principles of effective service message design:

1. Define as few message formats as possible: This increases the odds that one service's output message is another service's input message
2. Make all request messages as simple as possible: Write your services so that, when processing a request message, they take all reasonable defaults and ignore "extra" information
3. Have response messages return as much data as possible: When designing a response message, think "business transaction document" rather than "third normal form data"

How the Principles Work
Once you've created enough services, consumers will start to put together those services to create new applications. This process of "orchestrating services" is easiest when operations on the various services share message formats. The ideal situation occurs when the output/response message from an operation on one service is exactly the message required to request the next operation in the orchestration.

You could, of course, work out in advance what sequences of operations will be used and define the output messages to match the required input messages. Unfortunately, predicting those combinations would be an awful lot of work and, more importantly, you'd get it wrong: In an environment with lots of services, clients will find ways to orchestrate services to create new application in ways that you can't imagine. Predicting the sequence of operations that will be assembled and coordinating their message formats is impossible.

The first principle provides an easier solution: Just have as few message formats as possible. Having very few message formats increases the odds that the response message from one operation is the request message for another operation. Having fewer message formats also makes life easier for developers because there's less to remember. Messages will change over time, but the fewer message formats you have to change, the lower your maintenance costs.

Of course, you can only get those benefits if you have a way of communicating your message formats effectively to developers. Having a tool that developers can use to check if their messages match your formats would be nice, also.

There is a tool that does all of that: JSON Schema.

Supporting the CRUD Activities
To demonstrate that claim, imagine a SalesOrder service that has implemented the basic CRUD activities (Create/add, Read, Update, Delete). Following the first principle, I'll have just one message format: a complete SalesOrder object. When requesting or deleting a single SalesOrder, a consumer would send a message consisting of just the OrderId. When updating a SalesOrder, the consumer would send an order with the changed properties filled in and include the OrderId; the message for adding a SalesOrder would require most of the properties, including the CustomerId, but the orderId would be omitted because that value would be provided by the service. The response message is the same for all of the operations: a complete salesOrder with all the properties filled in, representing the retrieved, added, updated or deleted order.

The JSON Schema in Listing 1 shows how to document this format in order to communicate it to developers. The real thing would be more complicated. I've only defined three properties for this SalesOrder object: customerId, orderId, and orderDate and haven't provided many constraints on those properties (for more on how to do that, see my earlier article on JSON Schema).

Listing 1: A JSON Schema for a Simple SalesOrder Object

{
  "title": "JSON Schema for sales orders",
  "id": "http://www.phvis.com/salesorder",
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",

  "salesOrder": {
      "type": "object",
      "properties": {
        "customerId": { "type": "string" },
        "orderId": { "type": "string" },
        "orderDate": { "type": "string" }
    }
}

It's easy to see how using this schema makes life easier for developers building consumers. To begin with, having a single message format means that the property holding the order Id is always called "orderId." Going forward, any changes to the salesOrder format only have to be made in this schema. As functionality is added to the sales order service consumers, additional properties can be added to this schema. Existing consumers who don't need that functionality won't need to be changed.

And, of course, creating new applications by combining services becomes easier when the output message from one operation is the input message for another operation. An application that confirms that a salesOrder exists before attempting to delete it would send a message with an orderId as a GET request and receive the full salesOrder as a response. That response message could then be sent to the service as a DELETE request to delete the salesOrder (following the second principle, the service would ignore the extra information).

Problems and Solutions
However, this schema isn't complete: The response message from the SalesOrder service will need to include some validation-related properties so the service can report problems with a request. At the very least, a numeric validationCode and a string validationMessage would be useful. A consumer sending a request to add or delete a salesOrder might need to include an operation code for anything beyond the basic CRUD activities.

It's tempting at this point to create multiple schemas: one for requesting reads/deletes and that consists only of the orderId, a second for updates/adds that contains all the saleOrder properties, a third response message that includes validation information, a fourth.... But this is madness: Simply ensuring that all the necessary property names were identical among all of the formats would be a pain. If you add another property to the salesOrder (or modify an existing one), you'd have multiple formats that you'd have to update. You'll have thrown away the benefits of a single message format.

Fortunately, JSON Schema provides a simple way to extend my base definition using the allOf and $ref keywords. The first step in this process is to rewrite my original schema to move my salesOrder into a section called definitions as shown in Listing 2.

Listing 2: An Extendable salesOrder JSON Schema

{
  "title": "JSON Schema for sales orders",
  "id": "http://www.phvis.com/salesorder",
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",

  "definitions": {
    "salesOrder": {
      "type": "object",
      "properties": {
        "customerId": { "type": "string" },
        "orderId": { "type": "string" },
        "orderDate": { "type": "string" }
        

Now, to extend a schema, I create a new object definition that builds on my salesOrder schema by using the allOf and $ref keywords. The allOf keyword accepts an array of JSON schema definitions. Any JSON object that validates against the schema must be valid against all of the definitions. To include my base salesOrder object, I use $ref to refer to my SalesOrder schema in the definitions section. The example in Listing 3 defines a salesOrderResponse message by extending my salesOrder definition with some additional validation-related properties.

"definitions": {
    "salesOrder": {
      "type": "object",
      "properties": {
        "customerId": { "type": "string" },
        "orderId": { "type": "string" },
        "orderDate": { "type": "string" }
     }
   },

   "salesOrderResponse": {
      "allOf": [
         { "$ref": "#/definitions/salesOrder" },
         "properties":{
            "validationCode": { "type": "number" },
            "validationMessage": { "type": "string" }
         }
      ]
   }

While allOf and $ref provide some of the features we associated with object-oriented inheritance, it shouldn't be confused with inheritance. While the allOf keyword allows me to extend a schema, I can't use allOf to override the definition of orderId and, for example, specify that the property is now an integer.

I can, however, use allOf to create and extend my base salesOrder to add additional properties required in requests (an operation code) or, really, to provide any properties required by any message. As with the original schema, developers can use this schema to validate messages in the Visual Studio editor (as I discussed in that previous article) or to validate messages dynamically at run time.

This new schema isn't quite good enough, however. To make this base salesOrder format truly useful, I need to be able to specify which properties are required in each message type. I'll tackle that issue in my next column along with some additional tools for extending the base format.